1) Jar files.
--------------
The problem here is that people must first maybe install JRE, then install java3D, then download the files and lastly run them. Surely a bit too unconvinient for an "average user".
Thats why instalelrs were born. InstallAnywhere NOW is even free.
2) As applet.
----------------
It's maybe not the nicest solution for some bigger game, but why not, at least it starts immediately.
The java version is detected and a newer version of java is installed if needed, a java3D plugin is installed as well, all this automatically.
Um HOW are all these amazing things happeneing? Applets dont do any of this by themselves.
I think you are confusing Applets with Webstart.
3) With webstart
----------------------
first problem: no auto-install for java3D, must sign jars, make things... and each program will have their own copy of java3D! (well, i think, because they aren't installed as extensions but as native libs)
yes this is a problem. Webstart as we ship it will only get auto-installers from the Sun installer server which isnt very up to date

if yo uare up for soem coding you might be able to get around this by using OpenJNLP. The only question is how you get that JNLP client down first. Basically a bootstrap problem.
second problem: everything must be signed, again!
third problem: plugin architecture impossible! (Or not?!)
Im not sure but I believe webstart apps can request permissions without beign signed. then its up to the user if they want to trust your unsigned app...
Seems to me you left one possability out. Its the simplest. Simply tell your users they need to insatll the JRe and Java3D and give them links to downlaod the installrs. Not ideal I know but they will only need to do it once...