I've come across the same choice lately. It appeared that (as always) there is no definitive answer to this.
- Using Swing is a quick but a bit ineffective way of doing the job (for the moment). Perhaps Mustang will solve this (albeit only with the Sun JVM ?). I used the bridge existing in Xith3D it gives you a good example of how to do this (It is a bit buggy with the event processing).
- Using a texture for the complete screen was not adapted for my system (too high video memory & fill rate consumption) but I can imagine that some UI would benefit for it (when they cover most of the screen). I did not try this.
- Using a texture per top level widget seems to be a good trade-off since it allows you to only update the modified part of the texture and not send the whole widget like you would with glDrawPixel.
- For some widgets it happens to be usefull to be allowed to provide free geometry (example : in the player selection buttons of my game, you have an animated model of the selected character).
In my system, I end up using the main engine scenegraph for the UI part ;
- A top-level shell is in charge of managing the dispatch of events to top-level widgets as well as managing the laying-out updates,
- By default, each widget adds up and manages the geometry it wants to the top-level shell,
- Special top-level window allows for texture cached widgets (planned in the design but not yet implemented),
- The z components of the geometry define the z-ordering of the widgets.
- The UI is rendered in its own pass after clearing the depth buffer.
This allows the UI to use all the feature of the graphic engine (like animated models, correctly sorted blending,...).
This design looked to me like a good compromise between the performance cost and the design flexibility.
I think that you can come with lots of clean other solution for this.
For another solution, you may want to have a look to
http://www.cegui.org.uk/ which has proven to work for a few projects.
Vincent