In other worlds it seems that neither OrderedGroup neither DecalGroup is ever handled differently from just ordinary Group they extend...
Not true. I am using OrderedGroups extensively, and they work for me perfectly.
1. OrderedGroup works ONLY for Opaque shapes, and ONLY when opaque sorting policy set to View.OPAQUE_SORT_BY_SHADERS (should be default).
2. OrderedGroup DOES NOT WORK for Transparent Shapes (for now - can be changed afterwards)
3. On Opaque Shapes, you will see the result of OrderedGroup IF AND ONLY IF you disable the depth test - otherwise gfx card still may skip pixels because of they do not pass configuret depth test
4. It is very easy to add new sort modes, but it is not a good idea right now to let user set a custom state sorting function because of it will require exposing too much of rendering pipeline internals that may (and will) change in future.
5. As of rendering coplanar geometry, in OpenGL there is a special thing called Polygon Offset for handling such a cases - refer to Xith3DPolygonOffsetTest.java I wrote long time ago when added support for this feature. It also supposed to handle "nearly co-planar" shapes, as well as "exactly coplanar" which you are testing.
6. To check details of how state sorting works, explore Renderer.buildRenderFrame - it controls all the sorting.
7. Practically, you will get desired results (I think) when change depth test function in RenderingAttributes to LESS_OR_EQUAL instead of default LESS - this will allow one shape to override the other even if depth buffer already has equal value (which is not the case in default configuration).
8. Note that currently there is hard-coded limitation of 10 nested ordered groups (which is easy to change). Refer to render.OrderedState for details how it is built.
... I even think I missed something, but I believe that 1-1.5 years ago I already was describing behavior of ordered groups in details when transparency rendering was disucssed...
Its seems they all three are just equivalent...