to make the story short except in 1.6 update 10 there is no way to increase the JVM memory setting at runtime, and even a signed applet wont works. I found that the default heap size may vary depending on target JVM and maybe OS not sure.
one workaround could be to use streaming and allocate object only when they are requiered rather than allocating all objects when starting.
In theory, you can launch a separate java VM in your signed applet, and make both VMs map the same file (MappedByteBuffer). Now you can transfer data between the two VMs at full speed (capped by memory bandwidth).
Works in 1.4.2+ as opposed to 1.6.0_u10+