I'm not an expert in Java3D but these problems are common regardless of 3D API.
You'll get good scene antialiasing if you render your image in double size then scale the image down to the desired size.
Several years ago I used the following algorithm (if I remember it correctly
) to get an image with the alpha channel:
1. Render the scene on black background, lighting off, texturing off (actually I used alpha channel of textures as the color texture to keep the "holes"), geometry color is set to desired alpha value. Grab the result - you've got the alpha channel.
2. Render the scene as usual and grab the result - you've got the color channels.
3. Combine the channels into one image.
This algorithm combined with the scene antialiasing produces good result.
Hope this helps!