Like a way to view the size of the file, and if it is over the heap size, dont allow the user to upload the file?
You can use File.length() for that and the methods in Runtime.getRuntime().
Is there a way to call -Xmx from within the program, or does this have to be done at the batch file before the program starts?
You can't change the max heap space from within the program. You can only set it using the -Xmx VM argument.
I also came across an interesting article about this matter that might be of use:http://www.roseindia.net/javatutorials/OutOfMemoryError_Warning_System.shtml