UDP with a TCP control channel

[kevglass]

Now, I really don't want to start the UDP vs TCP argument again but..

I've been working on AstroPrime (http://www.cokeandcode.com/astroprime) which requires fast, low latency comms to describe the position of the ships. However, due the way the protocol works (collision reaction) I need the messages in order and with guarateed delivery.

The only way I can see to get this is to implement something on top of UDP (which incidently, I've done), but I'm getting all sorts of annoyances where I seem to lose a fair ammount of packets, which ties the thing up and eventually disconnects.

So... I was going to try a different implementation (no threading, all polling). I was going to use UDP to get the low latency messages to the other end and then use TCP to send ACKs back on the TCP stream.

Is this possible/done before? Is there a library I can pick up to do this sort of thing for me? Is there a better way?

I realise TCP with no nagles should be damn quick, but I still seem to get a fair bit of addition overhead when playing across the internet at large..

Any advice, suggestions, HELP! appreciated...

Kev

[Markus_Persson]

Well, I just sat down and  didn't let  myself  leave  my  computer until the udp package  never  skipped  a packet  (and packets were only processed once).

It resends  unacked packets after n ms,  and  if it hasn't gotten  a reply, it  just considers the connection timed  out and throws exceptions.


The only remaining issue right now is for network commands that rely on previous commands ("add backpack to inventory",  "add boomerang to backpack")..  I've got some ideas, but  just sorting all udp  packets would  just make  it TCP/IP  again.

[darcone]

I would use UDP for unimportant info as location update etc that doesnt have to be resent... Otherwise you will have to consider all problems that can happen using UDP as out-of-order packets, packet duplication, etc.. Which would leave TCP better in the end

Games published by our own members! Check 'em out!

[kevglass]

Well, yes, I'd originally though that. But I wrote a TCP like layer over the top of UDP (where I did resends etc) and kept ordering correct, and it performs far better than TCP simply because I have control of exactly when a datagram is sent.

However, this layer isn't very well written and is causing me a headache, hence the thoughts about a change..

Kev

[Markus_Persson]

Well, for a lot of data you  don't care about the orders.. you can usually just have an increasing id on the packets, and ignore any packets that's not the latest one for stuff like movement.

I think we might be able to make the protocol in Wurm totally asynchronous with just updating of states and keeping track of the id of the packet that last changed the state. (so if an older packet tries changing it back, you just ignore it)

[Athomas Goldberg]

However, this layer isn't very well written and is causing me a headache, hence the thoughts about a change..

The big question is whether you can completely eliminate these headaches without reimplementing TCP (or enough of it that you lose the performance gains).

I think we might be able to make the protocol in Wurm totally asynchronous with just updating of states and keeping track of the id of the packet that last changed the state. (so if an older packet tries changing it back, you just ignore it)

Interesting idea. If bits of state can be modified independently (position, current weapon, health, etc) all you need to do is keep seperate "most recent packet" ids for each.

[morbo]

Tribes 2 implemented a network model like this:

http://tork.zenkel.com/uploads/pdf/TribesNetworking.pdf

They broke network data down into different categories and handled each one differently, depending on the application needs. Seemed to work pretty darn good, too.

[Jeff]

Well, yes, I'd originally though that. But I wrote a TCP like layer over the top of UDP (where I did resends etc) and kept ordering correct, and it performs far better than TCP simply because I have control of exactly when a datagram is sent.

Hm. Not sure I follow you here.

You ARE disabling Nagle on your TCP tests, yes?

JK

[Jeff]

I think we might be able to make the protocol in Wurm totally asynchronous with just updating of states and keeping track of the id of the packet that last changed the state. (so if an older packet tries changing it back, you just ignore it)

Yep this is a common approach for asynchronous state communicating games.  You just send packets with the state of a given object at a given time , predict/interpolate for the time between that and the next packet, and ignore any out of sequence packets as arriving "too late."

The issue is when you have order critical events in your game.  At that point you need to write additional logic or fall back to TCP.

To answer the original question, at TEN we had a UDP/TCP hybrid protocol called BULLET our chief architect came up with for accelerating Quake comms.  Unfortunately I don't remember the exact details, I don't have his slides on it, and as he now runs EA Online he's unlikely to share it again

But I've seen evidence that you can indeed get better performance for at least some kinds of apps by using TCP and UDP in tandem in clever ways.

[Markus_Persson]

No, all packets in wurm are already guaranteed to arrive at one point, so this would work for critical events (deaths, level changes) as well. I think.

Games published by our own members! Check 'em out!

[blahblahblahh]

Now, I really don't want to start the UDP vs TCP argument again but..

However, due the way the protocol works (collision reaction) I need the messages in order and with guarateed delivery.

The only way I can see to get this is to implement something on top of UDP

This may be a silly question, but if you want all that (in order, guaranteed delivery) then what is there in TCP that you *don't* want? Where are you going to make any improvements in efficiency or speed?

W.r.t. any problems you've been having...if you find TCP is slow, but your hack on UDP is fast, my first guess would be that your MTU, MSS or similar is slightly too big, and you're getting your packets abnormally fragmented.

[Jeff]

And be sure you have Nagle disabled or you will see your small packets arriving in delayed clumps rather then one at a time.

[kevglass]

Ok, Ok, I'm sure I've got nagles turned off. And I still get slightly worse latency on TCP compared to UDP (with my additions), however...

My network stuff has suddenly become all nice, since I discovered how to use Selector in nio. I now just support both UDP and TCP and make the decision in the game logic as to whether its important that the messages gets there..

Astroprime now plays better and is way more stable.. new version soon.

Kev

[blahblahblahh]

My network stuff has suddenly become all nice, since I discovered how to use Selector in nio. I now just support both UDP and TCP and make the decision in the game logic as to whether its important that the messages gets there..

Given how much joy is spread amongst java developers by the extremely well-documented nio classes </sarcasm>, I'm sure many would be interested to see the relevant excerpt of your code .

Especially given that there is still AFAICS no documentation for UDP + nio. (I considered upgrading some old UDP production code to fit in better with NIO, but decided a working non-nio version was worth more than a possibly-working-but-no-one-can-really-be-sure hacked-together version using NIO ).

[kevglass]

An excerpt probably isn't going to help much but I'll do something about documenting what I currently have then make it available if ya like..

Still not sure its can be defined as "right" but it does seem to work ok..

Kev

[swpalmer]

This would be great stuff for the Wiki.

[kevglass]

Ok, I really haven't had time to document any of this, so I make the source available with the hope that it might help someone at some point.

http://www.newdawnsoftware.com/astroprime/net-stuff.zip

It contains a bunch of stuff I'm using for my network code:

Connection - Maintains a TCP and UDP (optionally) channel to send messages back and forth

ServerEndpoint - Handles receipt of remote connections

AbstractMessage - The abstract class describing all messages to be sent across connections

PollableGroup - The class that handles selectors.

Theres a test package in there that should still work to show how I'm using it. The source does create a lot of temporary objects, but I understand that isn't such a problem with the latest GC.

If its any help, let me know. If there are any glaring problem, let me know.

Cheers,

Kev

[Markus_Persson]

Quick sidenote: We changed to TCP from UDP in wurm, as the UDP package became more and more a "TCP over UDP" thing.. which is pretty much exactly what TCP is.


Pretty much ALL packages save movement is critical in an MMORPG, so I'm guessing the final implementation will use TCP for everything (~1.5KB / second, including terrain streaming) except character movement, and UDP (~0.5KB / second) for the moves..

[Herkules]

Good choice!! Same for me!

I hope to able to come up with a simple demo as well, soon.

[Jeff]

Quick sidenote: We changed to TCP from UDP in wurm, as the UDP package became more and more a "TCP over UDP" thing.. which is pretty much exactly what TCP is.
..

I've been making this argument for a  long time, glad to see a real world example

[endolf]

Hi
 The net stuff has both TCP and UDP in it because they each have their place, if you start writing tcp over udp you are using the wrong one. But as per the TCP v.s. UDP thread, there are reasons to use UDP, and with that in mind, the net code has both. Kev wrote this version of it, so i'm sure he'll agree

I'm doing almost exactly the same as Markus, everything but movement is TCP, movement is UDP.

Endolf

[Jeff]

No argument here. In fact there is even a usefulness for reliablity over UDP when you are doing reliable multi-cast.

The trick is to understand what each tool is good for and don't try to drive nails with your screwdriver

or to put it another way....

Addition is generally faster then multiplication.  But implementing multiplication ontop of addition is going to be a whole lot slower

[endolf]

don't try to drive nails with your screwdriver

Damn, I knew I was doing something wrong, explains all my BIY (Bodge it yerself) problems