Scenegraph serialization

[abies]

I would like to take a shot at it. There are few design decisions which needs to be solved first.

1) Do we use standard java serialization together with some trickery with read/writeObject and transient field or created totally own serialization protocol ?

Java serialization has main benefit of being fool proof at protocol level. It is also quite robust as far as future changes are concerned - can read old version of objects unless really major change happens. On the other hand, own serialization protocol would be a lot faster to read, plus it allows to do some tricks with sharing components between separately serialized branches.

2) How link nodes should be managed ?

To be honest, I have no idea if links are supported in xith3d at the moment, but anyway, it is something which needs to be thought about at this moment. I don't think that serializing nodes which are linked to is acceptable. I see two choices - either require user to manually reattach broken links, or provide some standarized way of managing/resolving references - probably through names. This way user would just have to put 'GenericTree' reference to some map. But the question is, if SharedGroup is referenced ONLY through links (as it probably should be), how can it be serialized ?  Explicitly ?


3) What about sharing components between separate streams ?
This is a harder case of above. What if some geometry components were shared (the same) in few different objects/parts of scenegraph and then serialized into different streams ? Should they be merged somehow ? From Paul Byrne's quote it seems that java3d scenegraph io somehow managed to do it:

There are also some key features of the SceneGraph IO API which you would loose by just using Serialization.

The main feature is that the API preserves the sharing of Node Components between BranchGraphs which are sent over the IO Stream in separate operations. This is not possible with serialization.....

4) Textures
Should they be serialized together with rest of stream ? Inline (in same file) or in separate file ? Using what compression ? (configurable?) Should there be a way to provide fallback for resolving textures from separate file anyway ?

5) Memmapped optimalization
In case of uncompressed textures and vertex data, it would be possible to align this data and use memory mapped file to directly use this buffer in GeomContainer (as long as it is read-only of course). This would allow to keep number of copies of same data down (1 at disk and 1 in GPU, instead of having another one in main memory).


We are talking only about preserving Locale/BranchGroup down. I'm not concerned about View/VirtualUniverse/etc - please say if you see any reasons for worrying about them.

[William Denniss]

Abies,

Firstly - I think this is an excellent step to take for Xith3D.  I have been using Serialization recentally in the network code of my game and it is a very powerful ally to have.

You have hit the nail on the head with the biggest problem - links and references post serialsation when sent in multiple operations and I am not sure the best way around it.

I'll describe the problem I had on the weekend.  For my game, when a client connects to the server - it recieves a snapshot of the game in progress, then delta frames.

I wanted the client however to pre-load all objects which are in use for the current map before it gets the snapshot (else it will be lagging from the start or would have to re-load every single possible object which is wasteful of memory).  The problem with that is once I sent an object containing a list of all the objects to load, I couldn't then send the snapshot as the references are lost.  Hence, read/wroteObject I found were not suitable.  So I had to do some custom post serialsation stuff to repair the links to the geometry which was loaded previously.

1)
I definitally think you should stick with java serialization to be compatable with the current tools.  Using transient is still viable without read/writeObject - but it's a little more tricky.  Perhaps you would need a Xith3D Serialization interface on top of Java's one to have some sort of post-serialisation trickery methods.

4)
I think textures probably should be transient - a nice way to serialize them seperatally would be convenient however.  that way people needing them can send them ahead of time.

I see no huge need for Views etc - if nessesary they can be broken down and reconstructed at the other end using some custom protocol.

Will.

[kimerinn]

Hey, guys, why we're inventing bike?
Scenegraph serialization is already invented, so it is no need to create something different (under 'serialization' I mean scenegraph-to-stream/file transformation, not just java serialization) . Look at the scenegraph.io package of Java3D (here are even sources there in j3d sdk). All problems with shared nodes, named objects, etc., are solved. All that needed is to port this machinery on xith3D.

Games published by our own members! Check 'em out!

[abies]

Java3d serialization is gigantic combination of writeInt/writeFloat code, 100-200kb at least. It supports many cases we don't care about, depends on some information we don't have, does not support some of extra stuff which was added in xith3d. So source porting is mostly out of question.

As far as taking ideas from it - sure, we will look at it. But there is no particular reason to follow their interfaces method for method, bit for bit.

[Yuri Vl. Gushchin]

100% agree with abies on this point.

I see serialization/deserialization in 3 places of Xith3D-based application development cycle:

1. Loading models from other modeling tools (covered by model loaders)

2. Model tuning in parallel with app development (scene graph/components serialization needed)

3. Final model deployment (I see here application-specific data formats or custom implementation of scenegraph serialization)

Yuri

[bmyers]

What about loading procedurally-generated models?

For example, we have a planetary scenegraph that is fairly complex, and takes about 30 minutes to generate from scratch.  We're currently using Java3D to serialize it to a file, and the deserialization to load it -- takes about 15 seconds to load the serialized version.

I'd love to be able to switch to Xith3D, but this is one of the big issues preventing me.

I don't need compatibility with the Java3D serialization, but I still need to be able to serialize/deserialize a given BranchGroup and all of it's subcomponents.

Would this fall in category #2 above?  

[Yuri Vl. Gushchin]

I don't need compatibility with the Java3D serialization, but I still need to be able to serialize/deserialize a given BranchGroup and all of it's subcomponents.  

Would this fall in category #2 above?

Yes, I think.

Generally my opinion is that scenegraph serialization should be easily detachable from rendering engine, similar ways to loaders. In fact, deserialization is nothing different than model loader, so only question is how to export specific model to specific file format.

Yuri

[abies]

I think we can make it separate. Just create mirror hierarchy of scenegraph classes in serialization code and let them do the job of serialization/deserialization.

There is a small difference between generic model loader and scenegraph serialization. Scenegraph serialization needs to be able to load/store classes unknown to it at design time (through some kind of interfaces to implement) and it should manage outside links. This makes implementation a bit harder than 'just another modelloader'.

IMHO it will be best to have some framework with default, all-encompassing serialization scheme. This will provide good fallback for people not wanting to mess with implementation themselves and at the same time guarantee that enough data is exposed to save/load state from code outside of main package.

I imagine some kind of generic serializer, which has control over stream as far as object references are concerned (to avoid duplicates), with collection of pluggable classes, one per concrete class in scenegraph, which will serialize/deserialize particular node/component. Everybody will be able to reimplement some/all of pluggable classes or add their own for custom subclasses, while keeping reference management part shared.

I'm NOT starting to work on it at the moment, so if anybody want to take a try, feel free, just say so, so we will avoid duplicated work.

[Yuri Vl. Gushchin]

There is a small difference between generic model loader and scenegraph serialization. Scenegraph serialization needs to be able to load/store classes unknown to it at design time (through some kind of interfaces to implement) and it should manage outside links. This makes implementation a bit harder than 'just another modelloader'.

Yes, 100% agree. This is quite similar to the problems that many 3D-authoring programs have with plug-ins.

With the rest, 100% agree with abies approach.

Yuri