Java-Gaming.org    
Featured games (81)
games approved by the League of Dukes
Games in Showcase (491)
Games in Android Showcase (112)
games submitted by our members
Games in WIP (556)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Questions about TCP sending and packet size  (Read 8001 times)
0 Members and 1 Guest are viewing this topic.
Offline roland
« Posted 2012-09-04 10:58:38 »

Hello, I had a problem with TCP. My networking code works fine when I run the server and client on my pc, but when my friend tries to join the game, or I join his, the data gets corrupted and the received length is wrong when a 22kb packet is sent in one go.

I changed it so I send the tcp data one byte at a time, flushing the output stream after each byte. I guess this is quite inefficient / slow? the main thing is that now it works.

So, what is optimal? and right now, I am waiting in an infinite loop for the next byte, like so (because I know how many bytes are expected, that was sent earlier), this is bad right? is there a better way? (read() blocks anyway, but I had to put it in a loop because each time flush is called i get a -1 on the receiving end)

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
private byte[] receiveData(int length) throws IOException
   {
      byte[] data = new byte[length];
     
      int offset = 0;
      while (offset < length)
      {
         int len = 1;
         
         
         int result = -1;
         do
         {
            result = is.read(data, offset, len);
         }
         while(result == -1);
         
         offset += len;
      }
     
     
     
      return data;
   }



Here's the server code

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
private void sendData(byte[] data)
   {
      try
      {
         int offset = 0;
         while (offset < data.length)
         {
           
            int len = 1;
           
            os.write(data,offset, len);
            os.flush();
            offset += len;
         }
         
      }
      catch (IOException e)
      {
         e.printStackTrace();
      }
   }


Thanks,
roland


Offline delt0r

JGO Knight


Medals: 27
Exp: 18 years


Computers can do that?


« Reply #1 - Posted 2012-09-04 14:26:28 »

First of all TCP is stream based. Not packet based. At least at the application layer which is what you care about.

Second after having worked on a lot of network code over the years in many settings, including java's tcp sockets, I can assure (99.99% sure) that there is a problem in your code somewhere to cause the corruption. The only outside 0.001% is some kind of crap router or wireless thing... but that seems unlikely since TCP sits on top of that and will resend and reorder data as required.

Flushing every byte is really really slow because TCP only looks stream based to you, the coder. Underneath it is packet based of course and each flush will result in a full IP packet which is IIRC bigger than 20 bytes. So you are expanding everything by a lot.  However your code does not look like you read one byte at a time. But up to length bytes.

So I would first of perhaps post your old code... we could see if there is an issue. Something does not seem quite right. I don't get -1 in my InputStream.read when i flush the other end for example. I may get a zero bytes read, but not a -1 (EOF/stream closed IIRC).


I have no special talents. I am only passionately curious.--Albert Einstein
Offline jonjava
« Reply #2 - Posted 2012-09-04 14:43:25 »

as delt0r says, TCP is stream based. (At least at the application layer which is what you care about.)

This means that you first establish and open a connection between two computers and a once you've done that you simply send and receive bytes from output and input streams of the socket connection. I.e, you don't have to worry about the other computers address after that, the two computers are essentially "linked" together.

UDP on the other hand is packet based or in other words "connectionless". This means that you need to put in the address and port inside every packet you send. (TCP fundamentally works like this as well, but it is abstracted from the application layer. TCP comes with other neat things like making sure packets arrive in the order they were sent and re-sending lost packets.)

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline teletubo
« League of Dukes »

JGO Ninja


Medals: 48
Projects: 4
Exp: 8 years



« Reply #3 - Posted 2012-09-04 14:53:03 »

As stated, you have to control the stream to check where a "message" starts and where it ends.

What I do in my game Reign of Rebels is to send a first int which will tell the size of the message, then read the message fully:
1  
2  
3  
4  
5  
6  
7  
   DataInputStream inFromServer;
...
    byte data[] =null;      
   
       int size =inFromServer.readInt();
       data = new byte[size];                    
       inFromServer.readFully(data);


note that if nothing is sent, the code will block in the readInt().

I wrapped a class to read and write primitives over network, if you're intersted, it works like this:


1  
2  
3  
4  
5  
6  
//Client sending message
PomPackage p = new PomPackage();
p.insert(35);
p.insert('c');
p.insert(new byte[30]);
sendPackage(p);


1  
2  
3  
4  
//Server reading the message:
int x = (Integer)p.pollQueue();
char c = (Character)p.pollQueue();
byte data[] = (byte[])p.pollQueue();


I might have reinvented the wheel but it works pretty nice, tell me if you're intersted.

Offline sproingie

JGO Kernel


Medals: 202



« Reply #4 - Posted 2012-09-04 17:10:40 »

If you do that sort of length-prefixed encoding, I recommend using some kind of framing sequence so that you can check for it and make sure you haven't gotten out of sync and started reading arbitrary bytes of messages and interpreting them as lengths.  If you stick a null byte before the length and after the message, you more or less have the websockets protocol.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 784
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #5 - Posted 2012-09-04 17:16:30 »

If you do that sort of length-prefixed encoding, I recommend using some kind of framing sequence so that you can check for it and make sure you haven't gotten out of sync and started reading arbitrary bytes of messages and interpreting them as lengths.  If you stick a null byte before the length and after the message, you more or less have the websockets protocol.
I don't quite agree with this. Encoding and decoding is so trivially simple that there is no way it will ever go out of sync.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline sproingie

JGO Kernel


Medals: 202



« Reply #6 - Posted 2012-09-04 17:17:41 »

there is no way it will ever go out of sync.

Gonna carve that on your tombstone are ya?  Wink

It's an extra byte.  Next time you make an off-by-one error, your blood pressure will thank you.


Offline Riven
« League of Dukes »

JGO Overlord


Medals: 784
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #7 - Posted 2012-09-04 17:19:02 »

there is no way it will ever go out of sync.
Gonna carve that on your tombstone are ya?  Wink
Let me just say that I never, ever, had my binary protocols go out of sync. persecutioncomplex


It's an extra byte.  Next time you make an off-by-one error, your blood pressure will thank you.
If you can't get 2-byte-length + payload right, and have to guard against off-by-one errors, you might as well stop coding. Don't get me wrong, I code extremely defensively, testing everything against everything, but not second guessing my own code to this extend.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online princec

JGO Kernel


Medals: 369
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #8 - Posted 2012-09-04 17:57:45 »

I feel a first time coming on!  Tongue

Cas Smiley

Offline teletubo
« League of Dukes »

JGO Ninja


Medals: 48
Projects: 4
Exp: 8 years



« Reply #9 - Posted 2012-09-04 18:18:42 »

If you do that sort of length-prefixed encoding, I recommend using some kind of framing sequence so that you can check for it and make sure you haven't gotten out of sync and started reading arbitrary bytes of messages and interpreting them as lengths.  If you stick a null byte before the length and after the message, you more or less have the websockets protocol.


It's easy to make sure you will not get out of synch when encapsulating a message. It's not like I will manually calculate the length of the message every time I send one.  And since TCP guarantees the correctness of the stream, I have to agree with Riven that there is no way it will ever go out of sync.

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline sproingie

JGO Kernel


Medals: 202



« Reply #10 - Posted 2012-09-04 18:35:24 »

I'm not talking about the stream being corrupted, I'm talking about reads going wrong, such as a short read not being handled, or some other programmer error that causes you to desync.  But I'm not going to die on this hill, so if that extra byte doth offend thee, pluck it out.  Hopefully your next layer up will at least detect the fault.

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 784
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #11 - Posted 2012-09-04 18:53:41 »

I'm not talking about the stream being corrupted, I'm talking about reads going wrong, such as a short read not being handled, or some other programmer error that causes you to desync.  But I'm not going to die on this hill, so if that extra byte doth offend thee, pluck it out.  Hopefully your next layer up will at least detect the fault.

If a mis-read of a int/short/float is allowed to corrupt the stream, then you're setting yourself up for disaster indeed. You have to guard against that, I do it like this:

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
// read the packet:
byte[] payload = new byte[dataInputStream.readUnsignedShort()];
dataInputStream.readFully(payload);

// handle the packet:
packetHandler = new DataInputStream(new ByteArrayInputStream(payload));
...
...
if(packetHandler.available() > 0) {
   // packet handler probably bugged
}


Note that this approach makes it 100% impossible to get out of sync, while using guard-zero-bytes only reduce the odds.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online princec

JGO Kernel


Medals: 369
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #12 - Posted 2012-09-04 19:07:19 »

While we're on the subject, TCP/IP's built-in error checking is not particularly robust - I've even had a data corruption on a LAN. Adding a CRC check to your data will make it near-as-dammit 100% robust.

Cas Smiley

Offline delt0r

JGO Knight


Medals: 27
Exp: 18 years


Computers can do that?


« Reply #13 - Posted 2012-09-04 19:53:19 »

Well IP packets have a 16 bit CRC header checksum IIRC . So that is no error on a bugged packet every ~16K packets *if* every packet is getting corrupted randomly. Only that is not really for the right layer....
 
The lower layers (physical/frame whatever) do a good job of not handing off bad packets in the first place. IIRC Ethernet is a 32 bit CRC checksum. Even with a lot of corrupted data, that is still a lot of packets between a missed corrupted packets. 

I have no special talents. I am only passionately curious.--Albert Einstein
Offline roland
« Reply #14 - Posted 2012-09-05 00:39:36 »

Thanks for the replies. My old version was simply:
1  
2  
3  
4  
5  
6  
private byte[] receiveData(int length) throws IOException //reading 22kb in one go, gets corrupted.
{
      byte[] data = new byte[length];
      is.read(data);
      return data;
}


1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
private void sendData(byte[] data) //sending 22kb in one go
{
      try
      {
            os.write(data);
            os.flush();
      }
      catch (IOException e)
      {
         e.printStackTrace();
      }
}


It may have been useful to say that I am also using UDP on a different port. I know there are conflicts so I should try to minimize UDP sending while downloading data with TCP. (When the player joins the server, TCP is only used to download the server data, then UDP is used for everything else).

Thanks for the information about flushing. I really want to make sure there is no chance of corruption, since TCP is supposed to be reliable...
I read up that the maximum packet size for ethernet is 1500 bytes, do I have to worry about this, or does TCP take care of it? eg. Should I write 1024 bytes at a time and flush after each?

Roland
Offline teletubo
« League of Dukes »

JGO Ninja


Medals: 48
Projects: 4
Exp: 8 years



« Reply #15 - Posted 2012-09-05 03:11:29 »


I read up that the maximum packet size for ethernet is 1500 bytes, do I have to worry about this, or does TCP take care of it? eg. Should I write 1024 bytes at a time and flush after each?

Roland

TCP takes care of it. Just make sure you read all the bytes you're expecting before stopping reading.

I recommend the approach I said in my last message: read an int (the size) then readFully(size). The first int will always be consistent, as well as the readFully (as long as your size is correct!), and with that you only need to flush once in the sender after the data is sent.

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 784
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #16 - Posted 2012-09-05 04:11:01 »

The problem of using an int for packet size (as opposed to a short) is that 1 client can hijack all available RAM of your process.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline jonjava
« Reply #17 - Posted 2012-09-05 06:16:28 »

The problem of using an int for packet size (as opposed to a short) is that 1 client can hijack all available RAM of your process.

Hmm?

Offline jonjava
« Reply #18 - Posted 2012-09-05 06:30:03 »

Oh now I get it!

[EDIT]: Also for a lot of things (simple things) a byte (255 max) length is fine.

To avoid checking mischievous packets you can add a small game id or give out unique  keys to your clients on connecting so you know where the packet is coming from. (At least mostly accident proof).


Offline roland
« Reply #19 - Posted 2012-09-05 06:52:20 »


I read up that the maximum packet size for ethernet is 1500 bytes, do I have to worry about this, or does TCP take care of it? eg. Should I write 1024 bytes at a time and flush after each?

Roland

TCP takes care of it. Just make sure you read all the bytes you're expecting before stopping reading.

I recommend the approach I said in my last message: read an int (the size) then readFully(size). The first int will always be consistent, as well as the readFully (as long as your size is correct!), and with that you only need to flush once in the sender after the data is sent.

Thanks Smiley that works perfectly, woot no more corrupted packets  Grin
I can't actually use a short / char because my packets will be larger than 65536 bytes, but this is only from server to client. If the client is sending I'll use this precaution.
Offline roland
« Reply #20 - Posted 2012-09-05 06:58:57 »

I wrapped a class to read and write primitives over network, if you're intersted, it works like this:

1  
2  
3  
4  
5  
6  
//Client sending message
PomPackage p = new PomPackage();
p.insert(35);
p.insert('c');
p.insert(new byte[30]);
sendPackage(p);


1  
2  
3  
4  
//Server reading the message:
int x = (Integer)p.pollQueue();
char c = (Character)p.pollQueue();
byte data[] = (byte[])p.pollQueue();


I might have reinvented the wheel but it works pretty nice, tell me if you're intersted.

I actually made something very like this for my UDP library Smiley
Thanks for the offer though!
Online princec

JGO Kernel


Medals: 369
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #21 - Posted 2012-09-05 08:55:23 »

The problem of using an int for packet size (as opposed to a short) is that 1 client can hijack all available RAM of your process.
Always always always check every single byte of data that comes over a network for validity. ints fine but enforce maximum size, etc.

Cas Smiley

Offline delt0r

JGO Knight


Medals: 27
Exp: 18 years


Computers can do that?


« Reply #22 - Posted 2012-09-05 09:07:50 »

Quote
1  
2  
3  
4  
5  
6  
private byte[] receiveData(int length) throws IOException //reading 22kb in one go, gets corrupted.
{
      byte[] data = new byte[length];
      is.read(data);
      return data;
}

Well there is your problem. is.read(byte[] data) is *not* is.readFully(). read(byte[] data) will block and read at least one byte or more, but does not block to  fill the  array, so you would often get partial fills, the rest would be old data in the array.

TCP really does take care of everything. Its just streams. But you still must use the streams correctly.

I have no special talents. I am only passionately curious.--Albert Einstein
Offline sproingie

JGO Kernel


Medals: 202



« Reply #23 - Posted 2012-09-05 18:18:55 »

And don't cross the streams.  Because that would be bad.   Grin
Offline ra4king

JGO Kernel


Medals: 345
Projects: 3
Exp: 5 years


I'm the King!


« Reply #24 - Posted 2012-09-05 21:30:40 »

And don't cross the streams.  Because that would be bad.   Grin
WIN!!! Grin Grin Grin

...... sorry for hijacking the thread ..... going back to hiding now persecutioncomplex

Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Nickropheliac (15 views)
2014-08-31 22:59:12

TehJavaDev (23 views)
2014-08-28 18:26:30

CopyableCougar4 (33 views)
2014-08-22 19:31:30

atombrot (41 views)
2014-08-19 09:29:53

Tekkerue (40 views)
2014-08-16 06:45:27

Tekkerue (35 views)
2014-08-16 06:22:17

Tekkerue (25 views)
2014-08-16 06:20:21

Tekkerue (37 views)
2014-08-16 06:12:11

Rayexar (72 views)
2014-08-11 02:49:23

BurntPizza (49 views)
2014-08-09 21:09:32
List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06

List of Learning Resources
by SilverTiger
2014-07-31 11:54:12

HotSpot Options
by dleskov
2014-07-08 01:59:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!