Java-Gaming.org Hi !
Featured games (83)
games approved by the League of Dukes
Games in Showcase (513)
Games in Android Showcase (120)
games submitted by our members
Games in WIP (577)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Kryo: Fast, efficient Java serialization  (Read 9731 times)
0 Members and 1 Guest are viewing this topic.
Offline Nate

JGO Kernel


Medals: 149
Projects: 4
Exp: 14 years


Esoteric Software


« Posted 2009-09-29 07:49:27 »

Project page:
http://code.google.com/p/kryo/

Comments, criticism, questions, let me have it. Smiley

I'm using Kryo as the basis for other projects such as:
http://code.google.com/p/kryonet/
http://code.google.com/p/legion/

Offline Nate

JGO Kernel


Medals: 149
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #1 - Posted 2012-04-16 20:34:37 »

Rewrote the library. Pretty sweet now!
https://github.com/NathanSweet/jvm-serializers/wiki/Unofficial-Results

Offline lhkbob

JGO Knight


Medals: 32



« Reply #2 - Posted 2012-04-16 22:47:12 »

Impressive, I'm definitely leaning towards Kryo when I get to the networking stage.

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline theagentd
« Reply #3 - Posted 2012-04-17 08:55:41 »

Impressive, I'm definitely leaning towards Kryo when I get to the networking stage.
+1

Myomyomyo.
Offline BoBear2681

JGO Coder


Medals: 19



« Reply #4 - Posted 2012-04-17 13:03:25 »

When do those results become "official?"   Tongue
Online atombrot
« Reply #5 - Posted 2012-04-17 14:53:56 »

I used kryonet for a small prototype (a networked snake) and its really easy to use. Only thing that bugged me was that you couldn't configure a dynamic buffer size. this led to some problems, as most messages I serialized were pretty small, but some on the other hand were pretty big (i wanted to add the possibility to transfer new maps and tilesets from server to client. I ended up not implementing the sharing functionality because I didn't want to set the buffer size to a very high value for all messages). I think the part with the buffer size lies in Kryo itself and not in kryonet...

Can the buffer size now be made dynamic? or can you propose a workaroun for this problem?

Apart from that I really liked working with kryo / kryonet, because it is very easy to learn and you can concentrate on other stuff than serialization or networking. Thanks man Wink
Offline gimbal

JGO Knight


Medals: 25



« Reply #6 - Posted 2012-04-17 15:31:04 »

kryonet looks really, really cool. Makes me want to add networking to my current game project Smiley

I wonder if it is not possible to modify the listener mechanism to bind it to a specific message type though, in stead of doing instanceof checks in the listener body to figure out what message type the listener received.
Offline Nate

JGO Kernel


Medals: 149
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #7 - Posted 2012-04-18 03:45:35 »

When do those results become "official?"   Tongue

Been waiting for the main project to pull updates from my fork. That's been done, and the "official" wiki page has been updated:
https://github.com/eishay/jvm-serializers/wiki

I used kryonet for a small prototype (a networked snake) and its really easy to use. Only thing that bugged me was that you couldn't configure a dynamic buffer size. this led to some problems, as most messages I serialized were pretty small, but some on the other hand were pretty big (i wanted to add the possibility to transfer new maps and tilesets from server to client. I ended up not implementing the sharing functionality because I didn't want to set the buffer size to a very high value for all messages). I think the part with the buffer size lies in Kryo itself and not in kryonet...

Can the buffer size now be made dynamic? or can you propose a workaroun for this problem?

Kryo now supports streaming, but KryoNet still doesn't. Currently KryoNet waits to receive enough bytes to deserialize, then deserializes. I don't think streaming deserialization for TCP makes much sense. Would the stream block until more bytes are received? That would mean the deserialization thread would block. Currently the deserialization is done on the network thread, which can't block as it is needed to receive more data. You could be receiving data from each connection. For a server this would mean a blocked deserialization thread for each connection. Fixed size buffers make this all much more manageable. Data is buffered until enough is received to deserialize.

What you really want isn't deserialization anyway, it's to send many small pieces and process them (probably write them to disk) as they are received. I do agree this should be easier.

I wonder if it is not possible to modify the listener mechanism to bind it to a specific message type though, in stead of doing instanceof checks in the listener body to figure out what message type the listener received.

You could create a listener that only handles a single message type, then add a listener for each message type. They will all get called when an object comes in, but only one would do anything, the rest would just return. You can also use Listener.ReflectionListener:

1  
2  
3  
4  
5  
6  
7  
8  
client.addListener(new Listener.ReflectionListener () {
   public void received (Connection connection, SomeClass object) {
      // ...
   }
   public void received (Connection connection, SomeOtherClass object) {
      // ...
   }
});


Add as many received methods as you like, the first method with a signature matching a received object will be called. You may like this more than many "if instanceof" statements.

Online atombrot
« Reply #8 - Posted 2012-04-18 07:15:17 »

Thanks for your reply. As I am quiet unexperienced with network programming, I thought there must be a reason for that constraint. Wink So what you say is that I have to serialize my data into one big stream and then split the stream in peaces that fit the buffersize and send it chunk by chunk? Maybe I could write a module to send and receive byte blocks and puts them together for further processing, when all chunks have been transmitted correctly.

This blocking sounds a bit like an issue to me, or at least something, I should keep in mind. What buffer size do you recommend to use, so the blocking won't be an issue? I will have to tweak it anyway, but would be nice to know what others are using here.
In my scenario, there would be one server and most likely not more then 8 clients. Most of the big data parts would be transmitted during setup of the game party, so in there it doesn't matter that much, if a connection gets blocked for a while (as long as it is not dropped). During the game, there will be transmitted most likely player events, world updates, chat messages etc.
Offline gimbal

JGO Knight


Medals: 25



« Reply #9 - Posted 2012-04-18 08:22:30 »

I wonder if it is not possible to modify the listener mechanism to bind it to a specific message type though, in stead of doing instanceof checks in the listener body to figure out what message type the listener received.

You could create a listener that only handles a single message type, then add a listener for each message type. They will all get called when an object comes in, but only one would do anything, the rest would just return. You can also use Listener.ReflectionListener:

1  
2  
3  
4  
5  
6  
7  
8  
client.addListener(new Listener.ReflectionListener () {
   public void received (Connection connection, SomeClass object) {
      // ...
   }
   public void received (Connection connection, SomeOtherClass object) {
      // ...
   }
});


Add as many received methods as you like, the first method with a signature matching a received object will be called. You may like this more than many "if instanceof" statements.

Neat! Perfect!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Nate

JGO Kernel


Medals: 149
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #10 - Posted 2012-04-18 08:53:46 »

So what you say is that I have to serialize my data into one big stream and then split the stream in peaces that fit the buffersize and send it chunk by chunk? Maybe I could write a module to send and receive byte blocks and puts them together for further processing, when all chunks have been transmitted correctly.

If your data is something like a file on disk, you can just send X bytes at a time until you've sent it all. If your data is a large object graph, then I guess you would send it in pieces rather than a single object. Or use a buffer big enough.

Quote
This blocking sounds a bit like an issue to me, or at least something, I should keep in mind. What buffer size do you recommend to use, so the blocking won't be an issue?

I was just describing how blocking deserialization won't work. KryoNet doesn't work this way.

I've added a method to Listener called "idle". It is called when a connection does not have much data buffered to be written to the socket. Then I added a class called TcpIdleSender and InputStreamSender. You use it like this...

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
InputStream input = ...;
// Send data in 512 byte chunks.
connection.addListener(new InputStreamSender(input, 512) {
   protected void start () {
      // Normally would send an object so the receiving side knows how to handle the chunks we are about to send.
      System.out.println("starting");
   }

   protected Object next (byte[] bytes) {
      System.out.println("sending " + bytes.length);
      return bytes; // Normally would wrap the byte[] with an object so the receiving side knows how to handle it.
   }
});


The listener added here removes itself when the end of the stream is reached.

Online atombrot
« Reply #11 - Posted 2012-04-18 09:15:06 »

Thank you very much for your explanation and example. I'll have to try that, when I find the time.
Offline Nate

JGO Kernel


Medals: 149
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #12 - Posted 2012-06-18 12:37:28 »

Just a bump to note the work I've been doing on Kryo lately (that is the serialization framework, KryoNet is Kryo + network stuff). The API and code is even cleaner since the Big Refactor mentioned above, and with more features too. Recently ReflectASM was updated to be able to access primitive fields without boxing, and now Kryo is faster because of it. Here are some charts to show the difference before these changes (2.15) and after (2.16):

https://github.com/NathanSweet/jvm-serializers/wiki/Kryo-2.15
https://github.com/NathanSweet/jvm-serializers/wiki/Kryo-2.16

Offline Cero
« Reply #13 - Posted 2012-09-19 19:38:41 »

I save a file with data.
I can load it too, works fine.

Now I change the class, I'm adding one field.

After that I try to load the file again.
Exception: Buffer underflow.

How do I write it so, that whenever a saved file doesn't have a field, but the class does, it just ignores it with a warning and the same for the other case: when a file has fields that the class doesnt have (anymore) ?

Offline Nate

JGO Kernel


Medals: 149
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #14 - Posted 2012-09-20 17:47:47 »

There are a few ways:
https://code.google.com/p/kryo/#Compatibility
I tend to use TaggedFieldSerializer.

Offline Cero
« Reply #15 - Posted 2012-09-21 17:29:16 »

yeah basically
1  
kryo.setDefaultSerializer(CompatibleFieldSerializer.class);

does exactly what I want.

My savegame class is like a struct anyway... no methods just data.

Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Longarmx (52 views)
2014-10-17 03:59:02

Norakomi (43 views)
2014-10-16 15:22:06

Norakomi (33 views)
2014-10-16 15:20:20

lcass (37 views)
2014-10-15 16:18:58

TehJavaDev (68 views)
2014-10-14 00:39:48

TehJavaDev (66 views)
2014-10-14 00:35:47

TehJavaDev (59 views)
2014-10-14 00:32:37

BurntPizza (73 views)
2014-10-11 23:24:42

BurntPizza (45 views)
2014-10-11 23:10:45

BurntPizza (85 views)
2014-10-11 22:30:10
Understanding relations between setOrigin, setScale and setPosition in libGdx
by mbabuskov
2014-10-09 22:35:00

Definite guide to supporting multiple device resolutions on Android (2014)
by mbabuskov
2014-10-02 22:36:02

List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!