Java-Gaming.org Hi !
Featured games (90)
games approved by the League of Dukes
Games in Showcase (741)
Games in Android Showcase (225)
games submitted by our members
Games in WIP (823)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Hotspot VM not deterministic under heavy load and swapping  (Read 5799 times)
0 Members and 1 Guest are viewing this topic.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1324
Projects: 4
Exp: 16 years


Hand over your head.


« Posted 2007-07-15 18:14:10 »

Scenario:
I'm giving the Wikipedia compression contest a try, where you have to compress 100MB of text to the absolute minimum. Not so much to try to win, but see what problems there are in this field.

As you can imagine, the program builds a lot of data-structures, pumps data among them, and analyses patterns in the words. It uses a lot of RAM, mainly in byte[], int[] and objects (no Buffers).

I can run the application a few times, and it crashes in a few kinds of errors. This is worrying, as there is only 1 thread working on the data, so IF there should be errors, it should be the same every time the program is run. Furthermore, the errors don't make any sense. Like copying from an int[] to an int[] with a for-loop, results in a ArrayStoreException, sometimes a NullPointerException, and a native crash (!) every once in a while, and 10% of the time the program runs just fine.

These errors start to occur when the program starts to swap to the harddisk / heap is completely flooded.

I tried both Java 5.0 and 6.0, same thing.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline keldon85

Senior Devvie


Medals: 1



« Reply #1 - Posted 2007-07-15 18:52:15 »

When dealing with a large amount of data you eventually have to start streaming using the Hard-Drive manually as VM (Virtual Memory) does not cut it. There may be existing libraries for this, but it's just basic caching principles anyway so it's not that hard to write.

Offline Riven
Administrator

« JGO Overlord »


Medals: 1324
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #2 - Posted 2007-07-15 18:56:22 »

AFAIK virtual memory is supposed to ehm... work??!

Even if it's factor 100 slower than doing things yourself (with a clever algorithm), it shouldn't corrupt anything, and certainly not make the JVM crash.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline keldon85

Senior Devvie


Medals: 1



« Reply #3 - Posted 2007-07-15 19:50:29 »

If memory allocation fails then programs are returned null pointers, so maybe Java is just built not to check this! Note that memory caching is not by any means difficult! You could code an error free implementation in under an hour, maybe even a couple of minutes!!!

What you should be getting is an OutOfMemoryException, so I'm not sure why you're getting such unrelated errors. Could it even be possible that you have faulty memory, or a faulty harddrive?

Offline Riven
Administrator

« JGO Overlord »


Medals: 1324
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #4 - Posted 2007-07-15 19:59:50 »

There's loads of virtual memory available. It's just that the Java Heap is full (and Java doesn't do any mallocs after the heap-size reaches its max).

It could be a faulty HDD, but hey, no other apps are crashing..

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline keldon85

Senior Devvie


Medals: 1



« Reply #5 - Posted 2007-07-15 21:00:09 »

Well you are in a situation where you know your program will exceed the default heap size, so the only options are to set the heapsize to be larger, or to create a solution using HDD caching. Whatever the answer to you question is, it is very unlikely to assist you in creating a solution that does not employ one of the choices I gave (unless there is another option).

You should be getting an OutOfMemoryException, so submit this as a bug to Sun and see what they come up with. Create a test case doing the most basic operations to duplicate this error using minimal code too so that you can be sure it is not down to anything else.

Offline Riven
Administrator

« JGO Overlord »


Medals: 1324
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #6 - Posted 2007-07-15 21:49:41 »

Pff... I'm not exceeding the default heap-size (that's impossible in Java). The heapsize is just to big to fit in physical RAM.

Further, I'm not out here for assistance, there is no answer to my question, because I didn't ask any. I'm posting here to report this unwanted behaviour.

I'll post a test-case later on, and no, it can't be 'down to anything else'. I'm using pure Java, so native crashes can't be blamed on my code.



It's realatively easy to code around it, but this shouldn't be happening!

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline keldon85

Senior Devvie


Medals: 1



« Reply #7 - Posted 2007-07-15 23:03:49 »

Oh right, well like I was saying you should report this to Sun. What I've read has said otherwise to the size of the Java heap (unless I'm reading something wrong).

http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_17470&sliceId=2
http://developer.apple.com/documentation/Java/Reference/Java14VMOptions/VM_Options/chapter_2_section_4.html

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
The heap is the area in memory in which objects are created.

    // Get current size of heap in bytes
    long heapSize = Runtime.getRuntime().totalMemory();
   
    // Get maximum size of heap in bytes. The heap cannot grow beyond this size.
    // Any attempt will result in an OutOfMemoryException.
    long heapMaxSize = Runtime.getRuntime().maxMemory();
   
    // Get amount of free memory within the heap in bytes. This size will increase
    // after garbage collection and decrease as new objects are created.
    long heapFreeSize = Runtime.getRuntime().freeMemory();

 - http://www.exampledepot.com/egs/java.lang/GetHeapSize.html

EDIT: Google terms were "finding java heap size"

Offline princec

« JGO Spiffy Duke »


Medals: 976
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #8 - Posted 2007-07-15 23:15:20 »

Riven - try running with -Xint so that compilation is turned off to see if it's the result of a compiler bug.

Cas Smiley

Offline moogie

JGO Ninja


Medals: 16
Projects: 6
Exp: 10 years


Java games rock!


« Reply #9 - Posted 2007-07-16 00:00:52 »

A little off topic, but how is your compressor attempt going?

Bewarned: it is a very addictive activity! I have been porgramming a wikipedia compressor off and on for months now. Every so often i find that i have an an idea which i must implement in the compressor ti find out whether it makes for better compression.

I really can't call me attempt a compressor per se... it is more of a pre and post processor which aims to convert the input xml file into a more readily compressable form and to use an existing proven compression technique for the actual compression.

My compressor is not very fast as i have favoured to develop it in a much easier to understand manner. Perhaps if i manage to make the compressor compress close or better than the record then i will attempt to optimise it to be fast....


Java4k RIP 2014
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline keldon85

Senior Devvie


Medals: 1



« Reply #10 - Posted 2007-07-16 00:04:59 »

Ok! I think I might join that obsession Smiley



Uh... Can I really modify a message that isn't mine? (me == Riven)

EDIT: It turns out you can Wink BY KELDON

8: Undefined index: online
File: /home/jgo/public_html/Themes/default/Display.template.php (main sub template - eval?)
Line: 161