check out
http://www.lwjgl.org for access to: Audio (OpenAL), Input, 3D graphics (OpenGL (which can be used for 2D graphics)). As for the math, I believe there are a few things in lwjgl for doing that sort of thing. You may also be interested in checking out JOGL, JOAL, and JInput if you are interested in seeing some alternative libraries.
Finished projects, thats sort of an in depth question, but the short answer is that you can use java webstart or an applet (depends on what your doing) or natively compile it, or use a native installer to hide the java stuff from your user and just let them interact with shortcuts.
Performance issues: read through some posts on the 'Performance Tuning' forum. If your worried about the garbage collector (seems like most people are afraid of that when considering java...) the key is to just not make tons of objects in every iteration of your main loop, theres no need to do it. There are also ways to tune the GC if it still ends up being a problem; you'll find info on this in the forum I mentioned previously.
if you are just starting opengl as well, then check out
http://www.nehe.gamedev.net and
http://java-game-lib.sourceforge.net/wiki/tiki-index.php when you are ready to start using it with lwjgl (if you choose to use lwjgl that is, otherwise I'm sure there are jogl resources around).
As for learning java, get a good book on the basics and then google and the java documentation will be your best friend from there

Good Luck!