You can certain set a different GC for applets (a few) and applications (all of the GCs included in the JRE). There are actually really good ones, and you really have to try which one works best for you. Just keep in mind that the more advanced GC normally have a lot shorter pauses, but have more overall
time spent in garbage collection.
With these parameters I get >5-8 collections per second of about ~3ms, in a small game producing rediculous amounts of garbage (by shooting bullets, explosions that mainly exist of thousands of individual pixels, where each pixel is a TransformNode in a scenegraph with Matrix4 instances for local and global transformations... etc etc) and giving each explosion it's own sound source.
I haven't tried GarbageFirst yet.