setEnableColorComponentWrite() works and 100% tested.
I used it in conjunction with custom build of JOGL (one that supports disabling swap buffers) to implement stereo glyph rendering (for red-blue and red-green glasses).
In your case, you just disabled writing to all color planes (R, G and B), so you will not see any visible changes anyway - only alpha writes are enabled, but they go to Alpha buffer, not to color buffer.
To see the effect, you have to enable Alpha buffer when creating CanvasPeerImpl by passing appropriate GLCapabilities with non-zero alpha buffer bitsize, then render with alpha write enabled, and use destination alpha value for something (blending, testing, etc.). But, AFAIK, there is still missing blending modes that support DST Alpha, I planned to add them but it was low-prio.
But when I write geometry with these rendering attributes basically nothing seems to come out...
Can you pls explain of what you are expecting to see?
P.S. Check also Xith3DColorComponentWriteTest - hit space and you'll see it works.