Java-Gaming.org    
Featured games (81)
games approved by the League of Dukes
Games in Showcase (495)
Games in Android Showcase (114)
games submitted by our members
Games in WIP (563)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Fast serialization  (Read 4413 times)
0 Members and 1 Guest are viewing this topic.
Offline loom_weaver

JGO Coder


Medals: 17



« Posted 2012-07-10 18:53:17 »

Quick question for you all.

I need to save/load a single zone as quickly as possible.  A zone may have up to 500x500 tiles representing the terrain.  Initially I was emitting XML for each tile and that was dog slow...

Now I'm investigating java.nio.* and I'm converging on serialization an array of 500x500x4 bytes in a ByteBuffer.  Most likely I'll keep the in-memory representation of the terrain the same as what's on disk.

My questions: is java.nio the way to go?  Any other thoughts surrounding this technique?
Offline sproingie

JGO Kernel


Medals: 202



« Reply #1 - Posted 2012-07-10 18:57:26 »

If you're planning on serializing a raw chunk of memory to disk, just go all the way and use a MappedByteBuffer
Offline Roquen
« Reply #2 - Posted 2012-07-10 19:26:08 »

Compression can sometimes be your friend.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline princec

JGO Kernel


Medals: 378
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #3 - Posted 2012-07-10 20:10:45 »

java.nio is probably not the best way to do this unless your data already naturally lives in a ByteBuffer for some reason.

Cas Smiley

Offline Roquen
« Reply #4 - Posted 2012-07-10 20:16:42 »

Using ANY binary format should drastically (and that's likely to be a big understatement) reduce the memory you need to move across slow buses and to/from a very slow device, so I wouldn't knock myself out with added solutions until you know where that gets you. 
Offline gouessej
« Reply #5 - Posted 2012-07-10 21:56:42 »

If you're planning on serializing a raw chunk of memory to disk, just go all the way and use a MappedByteBuffer

Riven showed me once a nice example using MappedByteBuffer, it might be helpful.

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 798
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #6 - Posted 2012-07-11 01:46:00 »

If you're planning on serializing a raw chunk of memory to disk, just go all the way and use a MappedByteBuffer

Riven showed me once a nice example using MappedByteBuffer, it might be helpful.
I searched the forum for a bit, to see what you were referring too. The only occasion where we talked about it, I showed you an example of mapping 200GB worth of data into a List<MappedByteBuffer>, which isn't very useful in this case.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Nate

JGO Kernel


Medals: 147
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #7 - Posted 2012-07-11 01:57:54 »

https://github.com/eishay/jvm-serializers/wiki/Staging-Results

Offline sproingie

JGO Kernel


Medals: 202



« Reply #8 - Posted 2012-07-11 04:11:00 »

Neat chart.  Any chance you could add JBoss Marshalling to the chart?

http://www.jboss.org/jbossmarshalling/
https://docs.jboss.org/author/display/JBMAR/Marshalling+API+quick+start
Offline Nate

JGO Kernel


Medals: 147
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #9 - Posted 2012-07-11 13:21:24 »

I hadn't seen that one before. I don't have the time at the moment to add it, but the jvm-serializers project has many contributors and is run by committee. If you push a fork on github that adds it, it would pulled (or merged or whatever the hell the git nerds arbitrarily decided to call things :p).

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline loom_weaver

JGO Coder


Medals: 17



« Reply #10 - Posted 2012-07-12 00:38:58 »

Thanks guys.  After refactoring my terrain data to be stored in a ByteBuffer and using java.nio for serialization I was able to cut the loading time of 500x500 tiles from 30 sec to under a second.  File size is much smaller too.

I may look into compression later on but I'm happy with the performance now and gzip'ing a data file reduces it down to about 1/3 the size.

Signing off with some Scala snippets.

1  
2  
3  
4  
5  
6  
7  
8  
9  
// save terrain binary data
val fos = new FileOutputStream(folder + File.separator + filename + ".dat")
val channel: FileChannel = fos.getChannel

gameState.terrain.data.rewind()

channel.write(gameState.terrain.data)
fos.close()
channel.close()


1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
// load terrain binary data
val fis = new FileInputStream(datFile)
val channel: FileChannel = fis.getChannel

val buffer = ByteBuffer.allocate(zoneData.width * zoneData.height * 4)

channel.read(buffer)
fis.close()
channel.close()

buffer.rewind()

Offline gimbal

JGO Knight


Medals: 25



« Reply #11 - Posted 2012-07-12 14:30:09 »

I may look into compression later on but I'm happy with the performance now and gzip'ing a data file reduces it down to about 1/3 the size.

Its 30x better. If you weren't happy with the performance now I'd stop by to give you a good kick to the bum.

Ah someone that took the plunge and actually uses Scala to do game dev. Might I pose a quick off-topic question and ask you what your experience is? Does it take away effort or is it more of the same using Java (the language)?
Offline loom_weaver

JGO Coder


Medals: 17



« Reply #12 - Posted 2012-07-12 18:38:26 »

Might I pose a quick off-topic question and ask you what your experience is? Does it take away effort or is it more of the same using Java (the language)?

I've been using Scala for about a year and a half now.  My IDE is IntelliJ combined with the Scala plugin.  I'm really liking the language.  At a minimum it's less verbose than Java but it allows you to use Java packages seamlessly.

The harder part is wrapping my head around functional programming and making the most of the various language constructs.  Took me about 8 months and taking a course by the Artima group before it really started to click.

I'm pretty much hooked now.  No going back to Java for me.
Offline erikd

JGO Ninja


Medals: 16
Projects: 4
Exp: 14 years


Maximumisness


« Reply #13 - Posted 2012-07-13 19:04:37 »

Quote
My IDE is IntelliJ combined with the Scala plugin.

How would you rate this setup compared to eclipse for java? One of the reasons I sticked with java is the tools, and I still think Eclipse for java pretty much rocks.
I'm interested in Scala though, but I wouldn't want to invest too much into it if the tools are not up there yet or if it is sort of an esoteric dead end.

As an example: I've invested a lot of time in Groovy after a big corporate push where I work, but in the end I found that it didn't help me at all. The tools pretty much sucked, and Groovy (even though it has some nice things) mostly seemed like smaller code was its main goal at the expense of conciseness, performance, and focus about what a language should be. It just seemed like a confused java knock-off with bad performance and some nice features that are implemented better in other languages.
Scala as a language seems to be much nicer though.

Offline sproingie

JGO Kernel


Medals: 202



« Reply #14 - Posted 2012-07-13 19:21:39 »

The IDE tooling for Scala has some pretty awful bugs in both IDEs that support it (Eclipse and IDEA).  Eclipse's support seems to be evolving at a much faster rate than the IDEA plugin, but when it breaks, it breaks eclipse-style, by popping up errors, requiring reloads, that sort of thing, whereas with IDEA it's more polished and much more occasional with breakage, only annoyances like pasting into the project always coming out as java and not scala.

Feature-wise they seem to be mostly at parity, both of them can do things like hilight implicit conversions and navigate to implicits in scope.  IDEA of course has a richer set of inspections.  Compiling on both still sucks, but IDEA has better fsc integration.  I'd still recommend just keeping a terminal window running sbt ~compile though (but eclipse does horribly unless you recompile now and then).  Both of them will occasionally red-line perfectly good code, and while you'd think Eclipse would do a better job since it uses the actual scala compiler, it sometimes baffingly redlines almost entire files for no reason til you force a recompile, whereas with IDEA it's more a matter of the inference in the type-aware hilighting being weak (and that's steadily improving).

You definitely have to like the language enough to get over the crappy tooling.
Offline Nate

JGO Kernel


Medals: 147
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #15 - Posted 2012-07-13 21:33:32 »

As an example: I've invested a lot of time in Groovy after a big corporate push where I work, but in the end I found that it didn't help me at all. The tools pretty much sucked, and Groovy (even though it has some nice things) mostly seemed like smaller code was its main goal at the expense of conciseness, performance, and focus about what a language should be. It just seemed like a confused java knock-off with bad performance and some nice features that are implemented better in other languages.
+1

Offline davedes
« Reply #16 - Posted 2012-07-13 21:47:15 »

I've been working a lot with Python lately... Makes it really hard to go back to Java. Shocked

Offline gimbal

JGO Knight


Medals: 25



« Reply #17 - Posted 2012-07-17 12:31:04 »

Lets not derail the thread any further by yanking it out of the JVM environment. There are plenty of threads that deal with platform-penis comparisons.
Offline sproingie

JGO Kernel


Medals: 202



« Reply #18 - Posted 2012-07-17 18:33:48 »

Well there's Jython ... considering the thread was originally about serialization formats, the rails don't seem to be anywhere in sight.
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Dwinin (25 views)
2014-09-12 09:08:26

Norakomi (56 views)
2014-09-10 13:57:51

TehJavaDev (69 views)
2014-09-10 06:39:09

Tekkerue (35 views)
2014-09-09 02:24:56

mitcheeb (56 views)
2014-09-08 06:06:29

BurntPizza (40 views)
2014-09-07 01:13:42

Longarmx (26 views)
2014-09-07 01:12:14

Longarmx (33 views)
2014-09-07 01:11:22

Longarmx (31 views)
2014-09-07 01:10:19

mitcheeb (39 views)
2014-09-04 23:08:59
List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06

List of Learning Resources
by SilverTiger
2014-07-31 11:54:12

HotSpot Options
by dleskov
2014-07-08 01:59:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!