networking libraries

[bitshit]

Hi all!

Im going to do a project for school in which i'll do some research on various client synchronisation / prediction techniques. Since i'll (try) to implement various demo's demonstrating the effect using different network models (client/server, peer to peer) and protocols (tcp/ip & udp, i'm searching for a generic library that saves me from developing the network communication part.

Below are some libraries I found, but would like some feedback on:

Mina: http://directory.apache.org/subprojects/network/features.html
The feature list sounds complete, but Im wondering about it's performance (wont all these abstractions and generic handlers make things slower?) Does anyone have expirience using it in games?

JNAG: https://jnag.dev.java.net
The description does sound very usefull, but it doesn't seem to be a very active project 

JEnet:
I like the idea of being able to send some reliable packets, but when I would choose to use Mina there's probably no easy way to combine Mina and JEnet?

ODE networking: https://odenetworking.dev.java.net/
Im also interested in looking at this library, as it involves synchronizing object(physics) states over clients, which is something I want to investigate too. However the project space is empty. Neither can I access the SVN repository the author linked in this message: http://www.java-gaming.org/forums/index.php?topic=10733.0

Anyone got any expirience / thoughts to share on this?

Thanks!

Martijn

[bitshit]

Anyone 

[Jeff]

My only comment is that socket code really isnt that hard.  **shrug**

And if you are planning onm comapring techniques do you *really* want unknown processing handling in the middle?

Games published by our own members! Check 'em out!

[noblemaster]

MINA is cool! It uses NIO as underlaying protocol. Seems pretty fast, though I didn't
do a speed test.

[bitshit]

My only comment is that socket code really isnt that hard.  **shrug**

And if you are planning onm comapring techniques do you *really* want unknown processing handling in the middle?

Yes you're right, I'll try to implement my own nio server then. The real work will be in the game protocol and how to sync states... i think..

[karmaGfa]

JNAG: https://jnag.dev.java.net
The description does sound very usefull, but it doesn't seem to be a very active project 

Sorry for the delay. JNAG is involving slowly because I do some other things at the same time.
I wouldn' t recommend you to use it until it is more developed.

[Sakazaki]

MINA is cool! It uses NIO as underlaying protocol. Seems pretty fast, though I didn't
do a speed test.

I also use MINA in some little project.
It's an interesting library if you would like to build a custom protocol layer over TCP/IP.
Actually I can say it have some limits due to the fact that isn't so stable; documentations isn't clear (sometimes is simply too old) and there isn't a "state framework" that help to build a statefull client/server interaction (not a problem if you don't need it... and nothing soo difficult to write).
Do not seems to have speed issue (of course, I use it only in little environments).
I think is a good idea to take a look about it.

[sunsett]

Check out this post:

http://www.java-gaming.org/forums/index.php?topic=12384.0

-Matt Hicks

[sunsett]

Also, the ODENetworking project has been abandoned in favor of jME-Physics Networking (which is available for download at the JGN project as well).  I originally created ODENetworking to create an abstract physics networking API for use with any games that utilize ODE but since I develop exclusively on jME and jME-Physics has some proprietary changes to ODE internally it wasn't possible to do this and remain compatability with jME-Physics.  I have been seeking someone to take over the ODENetworking project and port jME-Physics Networking to ODENetworking, it technically shouldn't take too much effort, I'm just not willing to take the time to do it.

-Matt Hicks

[Herkules]

I'd like to advertise my HeadQuarter system a bit. (http://cvs.sourceforge.net/viewcvs.py/drts/projects/headquarter/).

Here are some facts

• based on NIO
• binary message format
• send, receive and distribute any kind of message (package objectbus)
• transparently maintain shared states (spatial data, properties, hierarchies: types/groups/...)
• clear idea of shared 'identity'
• notion of 'realtime'
• services like clock synchronization
• currently actively developed
• BSDL

Games published by our own members! Check 'em out!

[sunsett]

Herkules,

No offense, but looking at that API it looks incredibly more complicated.  I intend to look into it further though.

I would be very interested in some constructive feedback on my API as it's still pretty new and being developed.

-Matt Hicks

[Herkules]

No offense. But let me learn. What do you consider to be complicated?

[sunsett]

Sorry, I really wasn't trying to offend, I just came in to offer an alternative and respond with some constructive criticism.  I'm happy to explain:

Your SimpleChatClient for example, I kind of ripped some code but I think I've got the basic gist of it here:

1   2   3   4   5   6   7   8   9   10   11   12

NetStation mStation = new NetStation(); BusLine mLine = mStation.createLine(address, port); ChatClient mChat = new ChatClient(mStation, chatId); mChat.addListener(new ChatListener() {    public void newMessage(ChatMessage msg) {       System.out.println(msg.getSenderName() + ": " + msg.getMessage());       mChat.clear();    } }); mChat.send(mChatName, "Hello Chat!"); mStation.flush();

That to me seems a few too many things to keep track of.  Further, the ChatClient is part of the project itself, so assuming this was a custom implementation where they were creating this from scratch it would also need something similar to the ChatClient  implementation I'm assuming:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80   81   82   83   84   85   86   87   88   89   90   91   92   93   94   95   96   97   98   99   100   101   102   103   104   105   106   107   108   109   110   111   112   113   114   115   116   117   118   119   120   121   122   123   124   125   126   127   128   129   130   131   132   133   134   135   136   137   138   139   140   141

public class ChatClient {     private final BusStation            mStation;        private final Identity            mProtocolID;     private final ChatReceiver          mReceiver           = new ChatReceiver( this );     private final ArrayList             mListeners          = new ArrayList();     private final BusTicket             mTicket;             /**      * Creates a new instance of Chat.      */     public ChatClient( BusStation station, Identity protocolID )     {         mStation   = station;       mProtocolID   = protocolID;       mTicket      = new BusTicket( mProtocolID );       mStation.add( mProtocolID, mReceiver );     }         /**      * Shutdown the chat.      */     public void close()     {         mStation.remove( mProtocolID, mReceiver );     }          /**      * Get this list of messages that came in after latest <code>clear()</code>.      */     public Iterator getIncoming()     {         return mReceiver.getIncoming();     }             /**      * Release list of incoming messages.      */     public void clear()     {         mReceiver.clear();     }        /**     * Get number of incoming messages.     */     int getMessageCount()     {         return mReceiver.getMessageCount();     }            /**     * Retrieve a certain message.     */     ChatMessage getMessage( int idx )     {         return mReceiver.getMessage( idx );     }             /**      * Send message anonymously.      */     public void send( String text )     {         BusTicket ticket = createTicket( ChatConstants.SENDER_IS_ANONYMOUS );             ticket.putString( text );         sendTicket( ticket );     }             /**      * Send message giving the name of the sender.      */     public void send( String sender, String text )     {         BusTicket ticket = createTicket( ChatConstants.SENDER_IS_NAME );         ticket.putString( sender );         ticket.putString( text );         sendTicket( ticket );     }         /**      * Send message giving the <code>Identity</code> of the sender.      */     public void send( Identity sender, String text )     {         BusTicket ticket = createTicket( ChatConstants.SENDER_IS_ID );         ticket.putID( sender );         ticket.putString( text );         sendTicket( ticket );     }     private final BusTicket createTicket( byte sendertype )     {         mTicket.clear();         mTicket.putByte( sendertype );         return mTicket;     }         private final void sendTicket( BusTicket ticket )     {       mStation.broadcast( ticket, null );     }             // ---------------------------------------------------------------------------------------------    //    // Listener handling    //    // ---------------------------------------------------------------------------------------------         public void addListener( ChatListener l )     {         mListeners.add( l );     }         public void removeListener( ChatListener l )     {         mListeners.remove( l );     }         void notifyNewMessage( ChatMessage msg )     {         int cnt = mListeners.size();         for ( int i = cnt-1; i >= 0; i-- )         {             ((ChatListener)mListeners.get( i )).newMessage( msg );         }     }         }

Looks like complication to me. :-p

Here's an example using JGN for the client aspect assuming there is a server to respond (I make the same assumption for yours):

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18

NetworkingClient client = new NetworkingClient(); if (client.connectAndWait(address, port)) {     Thread t = new Thread(client);    // This is simply for convenience, in most cases you would call client.update() in your update method instead - it handles sending Noop to the server to let it know its still connected    t.start();     client.addMessageListener(new MessageListener()) {         public void messageReceived(Message message) {}         public void messageReceived(ChatMessage message) {             System.out.println(message.getSenderName() + ": " + message.getText());         }     });     // Send a message    ChatMessage message = new ChatMessage();     message.setSenderName("Me");     message.setText("Hello Chat!");     client.sendToServer(message); }

EDIT: Oops, forgot to include the best part, the ChatMessage:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20

public class ChatMessage extends CertifiedMessage {     private String senderName;     private String text;     public void setSenderName(String senderName) {         this.senderName = senderName;     }     public String getSenderName() {         return senderName;     }     public void setText(String text) {         this.text = text;     }     public String getText() {         return text;     } }

Cuts down on the learning curve when it's straight-up beans you have to create.

-Matt Hicks

[Herkules]

Ok, I see the point.

There is some boilerplate code for these reasons:

• This chat implementation can maintain any number of chat channels. e,g. 1 working C/S sending messages to all and another P2P. This needs some setup.
• The chat client buffers messages for most client won't be able to deal with asynchronously arriving chat messages.
• chat message senders can be anonymous, a string or an Identity (preferred, who says there has to be a name?). I'd normally assume that a players name is stored as a shared property in the Property subsystem and does not need to be transmitted with every single message.
• Networking or network message formats don't shine thru to the application. In fact, a HQ application doesn't really know wether it works across JVM boundaries or not.

So the (application) code to compare is:

1	mChat.send(mChatName, "Hello Chat!");

against

1   2   3   4

ChatMessage message = new ChatMessage();     message.setSenderName("Me");     message.setText("Hello Chat!");     client.sendToServer(message);

The long code you cite basically corresponds to your source of 'ChatMessage' plus some bound container buffering messages.

[colesbury]

Does Headquarters support UDP or just TCP?

-Sam

[Herkules]

Does Headquarters support UDP or just TCP?

HQ is TCP only for performance reasons.

Until now, there is no message that may get lost and order is important (or at leas useful)..

HQ heads for minimal network traffic which almost always requires reliability. This issues are discussed in other threads I think.
Implementations using other protocols are possible but currently not on the @todo list.

[sunsett]

How ironic, JGN is entirely UDP for performance reasons. :-p

UDP is significantly faster than TCP and much lighter.  I agree, the disadvantage to it is not having guaranteed delivery and not guaranteed order.  However, I have resolved these problems with CertifiedMessage (guaranteed delivery over UDP), and OrderedMessage (which extends CertifiedMessage and also guarantees the order of delivery).

As for your statement of my creation of ChatMessage being much longer than yours, that is just not true.  You have one line of code there, but under the hood you do this:

1   2   3

BusTicket ticket = createTicket( ChatConstants.SENDER_IS_ANONYMOUS );           ticket.putString( text );         sendTicket( ticket );

If I were to count (which I have been known to do on occasion - hehe) I think I come up with the same number.   Sure, I could create a little static method in ChatMessage sendMessage(String sender, String text) and that would offer the same benefit.  I wrote it this way to show how much more simplistic utilizing JGN in the same example is.

I'm not looking for competition though, I'm looking for cooperation.  My hope is that you'll see the advantages of JGN and hopefully want to help out.  I have no desire to maintain the project all by myself and the goal is to provide a standard that the community can rely on.

-Matt Hicks

[Herkules]

I just didn't dare to create a reliability protocol on my own. I was afraid of having to send too many UDP packets forth and back destroying the UDP advantage quickly while not being able to profit from vJ compression.

So HQ could operate with CertifiedMessage. But as long as TCP performs well esp. on thin-line scenarios....

[sunsett]

Well, I created JGN and PhysicsNetworking (an extension of JGN for physics synchronization in games) primarily focused towards first-person shooter style games where speed is key.

I have been considering adding the ability in JGN to use TCP as well, I just haven't had a need since the extensions I've written on UDP seem to fill all of my gaps.  The real place where UDP shines is when you're sending messages that are real-time messages like, "this player is in this location with this rotation".  I also introduced something called a PerishableMessage that is not guaranteed delivery.  In fact, if it doesn't get to its destination in a timely manner it gets rejected entirely.  The whole send and forget ideology works extremely well with UDP.  This is particularly useful for my PhysicsNetworking that sends PhysicsMessages that simply get sent 10 times per second and if one gets lost it doesn't matter because there's another one right behind it more correct anyway.

I did quite a bit of research into this when I first started looking at designing an API and the reason I decided to write my own was because all the fast-paced games seem to use UDP and nearly every API in Java I've found uses TCP.

BTW, packet compression is on its way in the near future.

Features done or coming soon: http://www.captiveimagination.com:88/wiki/index.php/Features

-Matt Hicks

[Herkules]

The discussion slips into the UPDvsTCP direction again so I will not continue that after this final comment on HQ's philosophy here.

HQ won't send messages that could be dropped. The real place where TCP/reliability shines is when you're sending messages that are real-time messages like "this player is in this location with this rotation". HQ uses dead-reckoning to only send the bare minimal amount of data. For dead-reckoning, you compare the actual local position with an estimated remote position. In order to be able to do a valid estimation, guaranteed, timestamped message delivery is mandatory. This way, when moving in a predictable way, NOTHING is send over the line which in all situations is better than any however-optimized message format.
Sending 10xsec brutally always was a no-go for HQ for it overcommits any line introducing higher packetloss and additional latency. You just cannot serve even a handfull of objects (HQ doesn't have a term like 'player') on an ISDN line with an update rate of 10/sec. In FlyingGuns e.g., 1-2 updates/sec occur although the  dead-reckoning treshholds are much tighter than necessary for a feasable gameplay.

Message compression does not make much sense for HQ, for currently all message formats are densely packed and (in particular) not self-descriptive thereby avoiding any overhead. The clean notion of Identity prohibits redundency. vJ just drastically reduces TCP overhead (from 40 to 3 bytes or so) whereas it cannot do anything about UDP overhead (20bytes). Everything in HQ is heads for *not* sending data on high-level in favor of sending it effectively on low-level.

Ok now, for me this is the end of TCPvsUDP before blah³/persson kills us here for being off topic.

My original concern was to point to HQ as an information backbone that is able to deal with shared states over a network. (BTW, HQ does not NEED to work over a network, it can be used locally as well. When FlyingGuns is configured to hold the 'server' within one client, this client and server don't  even use loopback to communicate saving a lot of overhead again). HQ is totally agnostic of running C/S or P2P or both. Besides it's basic infrastructure, it offers proven out-of-the-box services like clock sync and some specific shared states. The basic design allows for many improvements (high-level protocols, range-based services, automatic routing, object ownership, access rights....) that pend to be implemented.

[sunsett]

Then I'd say I get a last rebut against your statements.   The packet size of TCP is much larger than that of a UDP message.  Further, your premise of "HQ won't send messages that could be dropped" in so much as TCP messages can be dropped as well, it's just the protocol handles the resending.  In UDP JGN handles that instead of the protocol.  It's probably not as efficient or as fast, but it works.

In JGN I can reliably send more than 2,000 messages per second, so 10/second per player isn't bad for me.   If you're talking about a first-person shooter you just have to update that frequently or you are no in sync with the server reliably.  Look at games like Quake, Unreal, Battlefield 2, etc. and I believe you'll find updates at least that frequently.  One of the features on its way is prioritized update messages that send updates to clients less frequently depending on distance from the object.  That is a feature explicitly part of the PhysicsNetworking.

Something I haven't yet added is clock sync, but it shouldn't be too difficult to add.  I would be interested in how you went about implementing this?

-Matt Hicks

[karmaGfa]

Further, your premise of "HQ won't send messages that could be dropped" in so much as TCP messages can be dropped as well, it's just the protocol handles the resending.

He probably meant that HQ cannot afford to lose any of the messages that he send in order to work.

My message to all of you, network lib authors :
Make Loveibs, not war.

[Herkules]

My message to all of you, network lib authors :
Make Loveibs, not war.

I think we are pretty kind for we are talking about religion

Honestly, TCPvsUDP is a war longing for years meanwhile, w/o any result. Both works, depending on specific sitiuations. The arguments are quite obvious and always the same.

This thread (I think) was meant to collect network libraries available. I just wanted to add HQ to the list.

[sunsett]

No grudge held on my part.   My API will stand or fall based on its merit.   I would say you'd be hard-pressed to find many first-person shooter games that use TCP for their primary communication though. :-p

I can always appreciate some good competition, but my goal in creating this API was to get collaboration so there isn't 50 mediocre networking libraries but just one really awesome one.

From all the research I've done UDP is the undisputed champion for game development. Hey, I'd be interested in a stress-test challenge.   Would that forever resolve the UDP dispute? :-p

Herkules, I think your API has some really good ideas in it.  I would say the biggest drawback to using it though is the learning curve it takes to understand all of those aspects.  It seems pretty flexible though, which is great, it's just creating a custom message seems to take a lot of code to make happen.  If you could resolve that I would see a lot more usefulness to your API.

-Matt Hicks

[Herkules]

Something I haven't yet added is clock sync, but it shouldn't be too difficult to add.  I would be interested in how you went about implementing this?

Can be complicated, but in HQ its done very simple (hence not exact). It measures the time a message needs to go to the timeserver and back and the answer contains the timestamp from the server when it sent the answer. When the client receives the answer from the server, it assumes current time now is servertimestamp + roundtrip/2 and adjust the clients clock accordingly.

The key in fact is not that protocol but having an adjustable clock   (package de.hardcode.time)

[sunsett]

I was planning on implementing a similar concept but have a time sync and then create a conversion value for when messages are sent to that remote machine.  So it would say, "the time is +500 ms on the remote machine, so take my current timestamp and add 500 milliseconds to it before sending".

-Matt Hicks