Well, it seems that every OpenGL based engine has the same problem, I don't know about Direct X, but the "problem" is related to the depth Z-buffer.
Yes, rendering the transaprent objects is tricky for both OpenGL and DirectX, especially if you want to corectly render intersecting semi-transparent polygons or self-intersecting surfaces.
Currently to render semi-transparent shapes in Xith3D, you have to
1. Ensure that they will render on Transparent rendering pass (for example, set BLENDED transparency mode for TransparencyAttributes)
2. Ensure that geometry sort is enabled
3. Disable depth buffer writes for transparent objects
4. Choose the transparency sorting policy (check source and JavaDoc for details on each sorting policy).
I enabled sorting on Transparency attributes, and finally I set the View transparency policy to:
This fixes the problem as long as the polygons don't intersect, and as long as they are not very close to each other. Xith will sort all the polygons in the scene BACK to FRONT, but this sometimes doesn't work as expected.
TRANSPARENCY_SORT_BOUNDING_SPHERE _AND_EYE_RAY_INTERSECTION is a best and reasonably fast implementation I found for geometry sorting. If you have a better idea for it, lets discuss it - would be really great if we can enhance the transparency rendering [one of the approaches is to use more complicated bounding shapes, but this degrades performance very much].
To completely solve this kind of problems, we should implement some of multipass/miltiphase techniques for transparency rendering, which implements per-fragment (per-pixel) depth sorting and works perfect in every case, including intersecting polygons and self-intersecting surfaces. The good example for such technique is depth peeling, but it works efficiently only on newer hardware and requires more rendering passes.