Maybe something like
public class RenderBin extends Vector<RenderAtom>
could be a good idea.
Well, the RenderBin shouldn't be a fully featured Vector. I much more thought of replacing the (dynamicised) array by a List implementation (like Vector or ArrayList) and make use of Collections.sort(). But everything inside the class. If we'd extend Vector with RenderBin, we'd have to take care of overriding all the modification methods like add(), so it is better to just have one addAtom() method like it is for now and have a private field of type List<RenderAtom> instead of RenderAtom.
I think the separation of RenderBucket and RenderAtom was intended to make it possible for the renderer to store additional information without loosing it, when the stored RenderAtom changes or is recreated/replaced in the lifetime of the scene. So it might be a good concept to reduce recalculation/resorting effort, but not properly utilized in the current code base. Dunno it it is better to throw them away (since they are apparently not used to archive this kind of performance gain) or to think about how they can be used to our advantage.
A RenderAtom is once created and attached to a Shape3D object and is never removed from it (except when the shape itself is wasted). Any time a shape has changed, these additional information is resetted. So these information is always fixed to exacty one and the same RenderAtom.
As far as I understand the initial thought behind xith, it was designed to create RenderAtoms/RenderBins for all kind of rendering stuff and OpenGL communication, so you don't have to call methods on "low level" components like Canvas3D directly to perform such tasks.
Canvas3D is not "low level". It even is the most high level class inside the render package. It even was part of the scenegraph package before I moved it into the render package. It defines the absolute high level abstraction interface to the window, that is rendered on.
Storing the background color in a RenderBin enables the scene graph to contain nodes to change it, that could be controlled by scene graph functionality like e.g. a Switch. Am I wrong here?
Well, good point. Don't know if anyone ever wants to change the background color by a Switch node. But it is a possibility one should not loose. You're right. To make it possible without the necessity of having this RenderSetupAtom for only the Background color, I suggest to have a much cleaner way: Write an interface named "Switchable", which has only one method: setEnabled(boolean). This interface is then implemented by the Node class and the Switch makes use of it. Then you could write your own implementation of the Switchable interface to anything you like. The Switchable object will then not really by in the scenegraph like a Node would be, but it would be part of the scenegraph in this way.
I really like this idea and will write and commit it. I'll also write a first implementation of it (for the TK) for switching the background color.
Are the nodes aware of the RenderAtoms/RenderBins they are represented with? I always thought the two layers - scene graph and rendering - are strictly separated, so that the scene graph has no knowledge of the underlying rendering mechanism. Also the user should IMHO not be confronted with rendering specific classes while using the scene graph (from API-design perspecitve).
They're not "strictly" separated, but "mostly" separated. The Shape3D class has a field of type RenderAtom just like the RenderAtom holds a field of type <T extends Node>. So it actually is aware of the one attached RenderAtom, but doesn't make use of this knowledge (but it should do on an abstract way to increase performance). And it isn't aware of the RenderBins. Attaching the RenderAtom object to the Shape3D instance is necessary to not being forced to hold them in a Map<Shape3D, RenderAtom> and Map<RenderAtom, Shape3D> or something like that. It is a pragmatic and performand way.
The two packages cannot even be strictly separated. The scenegraph has very low knowledge of the renderer while the renderer has a very high one.
So moving the RenderBins to a scene graph class don't seem right to me.
For the multipass rendering just a better idea came to my mind. We/I could add a set of RenderPass objects to the Renderer class instance where each of them knows of a single instance of BranchGroup. This way the Renderer/scenegraph would be better separated in this point and the logic would even be better.
To get away from the each-frame-renderbin-refresh thing:
The there should be an interface called "NodeChangeListener" or something like that, of which an instance is added (or set. No need for a list of them!) to any Shape3D instance the Renderer will find. So the Renderer can update the RenderBinProvider in case of any change on the shape.
To further improve the performance for the case of several changes on the node at once there should be a flag to set by a beginTransaction/commit mechanism. First invoke the beginTransaction() method on the shape, then make changes while no RenderBin is updated, the invoke the commit() method and the RenderBinProvider is notified of the node change through the NodeChangeListener only once.
EDIT: Of course the Group class needs such a listener, too, to notify the listener of child add() or remove() (detatch).
The place in the BranchGroup offers another important advantage: When a modification on a shape has been done, we can move upwards in the scenegraph tree and find the root BranchGroup (if the shape is live) and make the necessary changes in the RenderBinProvider (which holds the 10 RenderBins), which is assotiated with the BranchGroup. When a Node is not live this can't be done and doesn't need to be done. When a Node is added to a parent, the modifiacation in the RenderBinProvider is done for the Node itself and possibly in it's subnodes if it is a Group.
Would that work in regard to the possible dynamic nodes/branches inside a scenegraph like again a Switch? Wouldn't that clutter the nodes with a lot of logic belonging to the renderer?
How would you for example take culling into account in this approach, so that culled renderbins are not updated?
I don't think a whole RenderBin is ever culled. I think I even know it, but I may be wrong. Will check it.