Fast serialization

[loom_weaver]

Quick question for you all.

I need to save/load a single zone as quickly as possible.  A zone may have up to 500x500 tiles representing the terrain.  Initially I was emitting XML for each tile and that was dog slow...

Now I'm investigating java.nio.* and I'm converging on serialization an array of 500x500x4 bytes in a ByteBuffer.  Most likely I'll keep the in-memory representation of the terrain the same as what's on disk.

My questions: is java.nio the way to go?  Any other thoughts surrounding this technique?

[sproingie]

If you're planning on serializing a raw chunk of memory to disk, just go all the way and use a MappedByteBuffer

[Roquen]

Compression can sometimes be your friend.

Games published by our own members! Check 'em out!

[princec]

java.nio is probably not the best way to do this unless your data already naturally lives in a ByteBuffer for some reason.

Cas

[Roquen]

Using ANY binary format should drastically (and that's likely to be a big understatement) reduce the memory you need to move across slow buses and to/from a very slow device, so I wouldn't knock myself out with added solutions until you know where that gets you. 

[gouessej]

If you're planning on serializing a raw chunk of memory to disk, just go all the way and use a MappedByteBuffer

Riven showed me once a nice example using MappedByteBuffer, it might be helpful.

[Riven]

If you're planning on serializing a raw chunk of memory to disk, just go all the way and use a MappedByteBuffer

Riven showed me once a nice example using MappedByteBuffer, it might be helpful.

I searched the forum for a bit, to see what you were referring too. The only occasion where we talked about it, I showed you an example of mapping 200GB worth of data into a List<MappedByteBuffer>, which isn't very useful in this case.

[Nate]

https://github.com/eishay/jvm-serializers/wiki/Staging-Results

[sproingie]

Neat chart.  Any chance you could add JBoss Marshalling to the chart?

http://www.jboss.org/jbossmarshalling/
https://docs.jboss.org/author/display/JBMAR/Marshalling+API+quick+start

[Nate]

I hadn't seen that one before. I don't have the time at the moment to add it, but the jvm-serializers project has many contributors and is run by committee. If you push a fork on github that adds it, it would pulled (or merged or whatever the hell the git nerds arbitrarily decided to call things :p).

Games published by our own members! Check 'em out!

[loom_weaver]

Thanks guys.  After refactoring my terrain data to be stored in a ByteBuffer and using java.nio for serialization I was able to cut the loading time of 500x500 tiles from 30 sec to under a second.  File size is much smaller too.

I may look into compression later on but I'm happy with the performance now and gzip'ing a data file reduces it down to about 1/3 the size.

Signing off with some Scala snippets.

1   2   3   4   5   6   7   8   9

// save terrain binary data val fos = new FileOutputStream(folder + File.separator + filename + ".dat") val channel: FileChannel = fos.getChannel gameState.terrain.data.rewind() channel.write(gameState.terrain.data) fos.close() channel.close()

1   2   3   4   5   6   7   8   9   10   11

// load terrain binary data val fis = new FileInputStream(datFile) val channel: FileChannel = fis.getChannel val buffer = ByteBuffer.allocate(zoneData.width * zoneData.height * 4) channel.read(buffer) fis.close() channel.close() buffer.rewind()

[gimbal]

I may look into compression later on but I'm happy with the performance now and gzip'ing a data file reduces it down to about 1/3 the size.

Its 30x better. If you weren't happy with the performance now I'd stop by to give you a good kick to the bum.

Ah someone that took the plunge and actually uses Scala to do game dev. Might I pose a quick off-topic question and ask you what your experience is? Does it take away effort or is it more of the same using Java (the language)?

[loom_weaver]

Might I pose a quick off-topic question and ask you what your experience is? Does it take away effort or is it more of the same using Java (the language)?

I've been using Scala for about a year and a half now.  My IDE is IntelliJ combined with the Scala plugin.  I'm really liking the language.  At a minimum it's less verbose than Java but it allows you to use Java packages seamlessly.

The harder part is wrapping my head around functional programming and making the most of the various language constructs.  Took me about 8 months and taking a course by the Artima group before it really started to click.

I'm pretty much hooked now.  No going back to Java for me.

[erikd]

My IDE is IntelliJ combined with the Scala plugin.

How would you rate this setup compared to eclipse for java? One of the reasons I sticked with java is the tools, and I still think Eclipse for java pretty much rocks.
I'm interested in Scala though, but I wouldn't want to invest too much into it if the tools are not up there yet or if it is sort of an esoteric dead end.

As an example: I've invested a lot of time in Groovy after a big corporate push where I work, but in the end I found that it didn't help me at all. The tools pretty much sucked, and Groovy (even though it has some nice things) mostly seemed like smaller code was its main goal at the expense of conciseness, performance, and focus about what a language should be. It just seemed like a confused java knock-off with bad performance and some nice features that are implemented better in other languages.
Scala as a language seems to be much nicer though.

[sproingie]

The IDE tooling for Scala has some pretty awful bugs in both IDEs that support it (Eclipse and IDEA).  Eclipse's support seems to be evolving at a much faster rate than the IDEA plugin, but when it breaks, it breaks eclipse-style, by popping up errors, requiring reloads, that sort of thing, whereas with IDEA it's more polished and much more occasional with breakage, only annoyances like pasting into the project always coming out as java and not scala.

Feature-wise they seem to be mostly at parity, both of them can do things like hilight implicit conversions and navigate to implicits in scope.  IDEA of course has a richer set of inspections.  Compiling on both still sucks, but IDEA has better fsc integration.  I'd still recommend just keeping a terminal window running sbt ~compile though (but eclipse does horribly unless you recompile now and then).  Both of them will occasionally red-line perfectly good code, and while you'd think Eclipse would do a better job since it uses the actual scala compiler, it sometimes baffingly redlines almost entire files for no reason til you force a recompile, whereas with IDEA it's more a matter of the inference in the type-aware hilighting being weak (and that's steadily improving).

You definitely have to like the language enough to get over the crappy tooling.

[Nate]

As an example: I've invested a lot of time in Groovy after a big corporate push where I work, but in the end I found that it didn't help me at all. The tools pretty much sucked, and Groovy (even though it has some nice things) mostly seemed like smaller code was its main goal at the expense of conciseness, performance, and focus about what a language should be. It just seemed like a confused java knock-off with bad performance and some nice features that are implemented better in other languages.

+1

[davedes]

I've been working a lot with Python lately... Makes it really hard to go back to Java.

[gimbal]

Lets not derail the thread any further by yanking it out of the JVM environment. There are plenty of threads that deal with platform-penis comparisons.

[sproingie]

Well there's Jython ... considering the thread was originally about serialization formats, the rails don't seem to be anywhere in sight.