Loading a 3D Map

[Conzar]

Greetings,
I have created a map loading system for my game; however, I have run into a memory issue.  Basically, the map that I designed in a 3D editor (AC3D) when loaded into my game produces an out of memory error.

I have a plugin for AC3D that creates the format for my game (it takes groups for specific map elements, writes them out as wrl files, and generates an xml file that associates the wrl file with the object type).

The map loader parses the xml file, creates game objects for the specific type and loads the wrl file for each object.

So what are the better ways of doing this?  I have many duplicated objects in the map (think cold coins from mario or pellets from pacman).  Right now I load each one separately.  Would it make sense to just load 1 wrl file and then use the Group.cloneNode() (in java3d) for every duplicate of the first one?  Will this conserve system memory or will it only conserve disk space?  The textures for each of these objects is just a reference in the wrl so there is just 1 texture file.

Thanks for your help.

[SwampChicken]

Well you've already hinted at a possible improvement.
Cache your world objects, so load one coin/pellet in memory and then replicate it around your world.

As for your world map, could it be a problem of your editor giving you too high a polygon count? For example, a hill consisting of 50 polygons going to be easier to deal with than the same hill modelled with 50 million polygons (but I'm sure you've already looked at that right?)

[Conzar]

Thats a good question.  So is there a general standard for the number of vertices or surfaces per map item/environment?

Games published by our own members! Check 'em out!

[anarchotron]

On a next-gen console or equivalent PC you're looking at rendering 100-500k textured polys per frame.  But that's only what's being rendered so you might assume your entire scene is substantially larger than that.  Each point holds position, normal, UVs, and maybe some other data like collision info (3+3+2+(say 3)=11 floats). At the low end 100k totally independent triangles will be 300k points or 3.3M floats, which is 13.2Mbytes of data.  Assuming you can index your triangles say you can cut the number of independent triangles in half.  So you're looking at 6.6Mbytes of data per view.  With a 60 degree horizontal FOV rotating the camera in place will net you about 6.6Mbyte*6=~40Mbyte worth of triangle data just in line of sight of your camera.  We haven't even counted up the texture memory.  There's a reason modern games typically run with multi-hundred megabyte heap sizes

(and stream in only what they need on the fly)

Just how much heap do you have allocated to your VM when you are getting the out of memory error?  Are you actually bumping up against your system memory limits or just the JVM heap limit?

[Conzar]

I had it set to 512MB and then bumped it up to 2GB and still got the memory error (this was done without any modification to the models or map loader).

Last night I did 2 things ... I reduced the surfaces from 1900 to 300 on the "coin" items that are repeatable.  I also copy those items when loading them in the map (so there is just 1 file that is loaded ... so this is an optimization for loading time!).

So it now loads up those items, but when the loader gets to another set of map type (spikes), the heap runs out of memory (with the heap set to 512MB).  The spikes are a type that can have a group of objects (multiple spikes) with different rotations.  So ithe map exporter just writes out each spike clump to a file and the map loader just loads in each spike clump.  I think the number of surfaces of the spike is about 400.  There might be about 200 - 300 single spikes in the map. 

Do u think that since the spikes are duplicated in the Modeler tool, that I should do the same thing as with the "coins"?  One big reason I have not done this with map elements like spikes is b/c getting the rotation information from the modeling tool is very hard  (You can't get it from any of their API b/c that information is impossible to calculate if you do not save the state of each rotation of the object; I've actually created a plugin that I can use to rotate objects in order to retain that rotation information, but its a pain to use ... its much easier to use the built in rotation tools instead of clicking on rotate buttons). 

Thanks for your help!

[anarchotron]

For static geometric objects you never want to duplicate the mesh in memory.  For the spike you just have a single spike mesh loaded in memory once and stored into opengl once (and then removed from your main memory if you can get away with it).  You are definitely taking the right approach.  There has to be a way to get the rotation data out of the map data (I'm not familiar with your map data at all though).

I don't know what your visual goals are, but the number of polys per item that you mention sound really high to me.  a 2k poly coin is pretty excessive   That's more than enough polys to model a good looking humanoid character, much less a coin.  300 polys is more reasonable but I would expect a 3d disc coin to be in the neighborhood of 50-100 polys depending on how many 'sides' your circles have.  400 polys for a spike sounds excessive.  When you say spike I think of a skinny cone, which could easily be modelled with 6 sides (6 polys!).  If the spike is all bendy or whatnot then of course it requires more.  These are the kinds of numbers I would expect for a realtime action game with lots of these objects all over the place.

With our theoretical 100k poly per frame target, you're looking at rendering about 160 coins and 125 spikes and nothing else. 

Using the instancing approach of duplicated geometry is how professional games get their poly counts up past 100k to the half million range per frame and still use reasonably detailed models.

[bahuman]

and now my two coins... cents 

The poly count sounds extremely high indeed, that's no doubt the reason you're running out of memory. Java comes with some basic post-mortem tools. See
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp#DebuggingOptions
http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html



Doesn't your 3D tool allow re-using models, or stamping or whatever? There has to be another way for your 3D tool to avoid duplication. Are you able to export extra information, like modelname, or materials, or even custom attributes from the 3D tool? In that case, you could use that extra information to fold all the models together again. For example, all meshes with material "spike" are ignored, and instead you replace it with your own mesh of a spike, that you re-use without duplication.

[Conzar]

Thanks for the debugging hints.  I actually use a great program called JProfiler (its not free but it provides many useful features).

As for the 3D tools.  I use a program called AC3D.  I wrote a plugin that exports the map to a format that my game understands.  So I was already able to do what you suggested with the "coin" objects; however, the spikes need rotational information.  So right now, the mesh is saved out in its final position which means there is no rotational information provided.  So I simply can't replace each spike mesh with a stored in memory spike b/c I don't have the rotation information.

I think the only way to do it is to update my rotation plugin ... I've just thought of a solution to 1 of the problems with it.  Basically, when you click on rotate on the object, there was a variable that is stored in memeory that keeps track of the rotation of the object.  If you export the object, then this information is saved in the exported file; however, if you just save the map in AC3D and then re-open it, that rotation information is gone.  I think I can actually save that information in a datastructure that AC3D api provides so that I can access it when I export it again.

But the other problem (bigger problem) is how do you know which order to apply the rotations?  Say u rotate along the X-Axis and then the Y-Axis.  Then later on you rotate again on the X-Axis.  It seems like one would need a list of rotations in the order in which they were applied to get the rotation down correctly right?  Initially I was thinking that you could just store the rotation information for x,y,z.  But the situation I described above nulifies that. 

So does anyone know how repeatable rotatable objects are handled in other games?

[anarchotron]

The easiest way is if you store your rotation data as either a Matrix or a Quaternion.  Euler angles are ambiguous because you may be unsure which order to apply the rotations.  Each time you rotation the object, just rotate the Matrix or Quaternion.  Then when you go to render the object you can either use your matrix/quaternion directly in your rendering engine, or you can always extract Euler angles from them and use those if you are more comfortable with that.

Basically the Matrix/Quaternion is just storing the final rotation of the object regardless of all the independent rotations leading up to the final state.

[Conzar]

From the lead dev from AC3D about this issue that I've raised in the past

this is not currently possible since the vertices are rotated in the global space.

And from an opinion of a user

It appears that a rotation is applied to the vertices then simply forgotten.

So from AC3D I don't think its possible to get the quaternion ... at least from their api.  So it looks like I would have to calculate the quaternion for rotations on my own.  Is there an easy way to do that?