for 2d the best hardware accelerated java library is
Slick2d, it should have everything you'd want from modern 2d library such as hardware accelerated graphics with OpenGL, faster sound with OpenAL, Controllers support, nice clean api, brilliant and helpful community behind it, plus a great set of tools such for fonts, game maps, particle editor, etc.
for 3d if you just want an engine/scenegraph then
jMonkeyEngine is a good choice. I'd avoid Java3D its been dead/abandoned for a while now and doesn't look like it'll get picked up again anytime soon.
another option is to go straight to the basics, with barebone libraries like JOGL+JOAL+JInput or all in one library
LWJGL, here you'll just get access to the raw api's of libraries like OpenGL, OpenAL and you'll have to manually role your own game library/classes to get going.