Java-Gaming.org Hi !
Featured games (83)
games approved by the League of Dukes
Games in Showcase (511)
Games in Android Showcase (119)
games submitted by our members
Games in WIP (577)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Partial Packets in NIO ?  (Read 4047 times)
0 Members and 1 Guest are viewing this topic.
Offline Endemoniada

Junior Newbie




Java games rock!


« Posted 2005-03-24 14:18:48 »

Hi guys,

How do you normally handle partial packets ?

Here is some pseudo-code:

if(key.isReadable()){
...
client.read(buffer);
...
// check if it's a full packet here ?
}

Should I be using something like XML tags, for example:

<in>the stuff he sent</in>

Then I can check it that way. And is it possible that the server can receive these kinds of packets:

// packet 01
<in>this stuff
// packet 02
wasn't complete</in>
// packet 03
<in>small</in><in>start of next
// packet 04
packet</in>

See what I mean ?

I'm pretty new to network programming but I'm a pro at C(++) so I'll get the hang of this stuff eventually.

Thanks.
Offline vrm

Junior Duke




where I should sign ?


« Reply #1 - Posted 2005-03-25 06:53:43 »

I use another trick, the first 4 byte of the packet are an Integer representing the total size of the packet, so I know how many bytes I need to wait.
Offline endolf

JGO Coder


Medals: 7
Exp: 15 years


Current project release date: sometime in 3003


« Reply #2 - Posted 2005-03-26 17:37:39 »

In my case I'm using simple messageing (enough for what I need) and I use the first 2 bytes as a message type code, and the next two as the data length, then I can read bytes off the stream till I've got that many and process them.

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Jeff

JGO Coder




Got any cats?


« Reply #3 - Posted 2005-04-16 01:44:32 »

This isn't really an NIO issue, its a TCP/IP issue.  TCP is a stream protocol, not a packet protocol, and as such doesnt necessarily return all the bits pushed into the stream at the same rate otu the other end.  UDP is just unreliable and therefor can split up or break packets.

While there are some knobs on TCP/Ip you cna twiddle and some assumptiosn you cna make for small UDP packets, the general rule is that you need to implement your own apcket protocol ontop of either.

This is what the guys were talking about with size headers-- they built simple protocols  that let them udnerstand wha they were receiving on the far end as descrete packets.

As I do packet communciation in my project the bottom-most layer of my app communciatio nstack is similar, an integer size proceeds each packet of data. NIO Buffers provide the scatter send methods that alloww yo uto build up protocol layers without having to copy data.

Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Offline William Denniss

JGO Coder


Projects: 2


Fire at will


« Reply #4 - Posted 2005-04-17 08:11:57 »

Jeff, I thought UDP did provide a pretty good (albiet unreliable) packet transmittion protocol already, the only reason I can see for creating your own packet protocol over the top would be if your datagrams were dangerously large (i.e. larger chance at getting lost) or if you needed some sort of reliability.  In which case you start to lose the benifits of UDP to begin with.

Will.

Offline Jeff

JGO Coder




Got any cats?


« Reply #5 - Posted 2005-04-18 04:06:22 »

Quote
Jeff, I thought UDP did provide a pretty good (albiet unreliable) packet transmittion protocol already,


I believe as part of that unreliability  UDP gives you no gaurantee that a packet will arrive in one peice (or even that all of it will arrive at all) but I'll double check as its been a bit...

Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Offline Jeff

JGO Coder




Got any cats?


« Reply #6 - Posted 2005-04-18 04:13:05 »

Okay ncie sumamry of UDP from here...

http://www-net.cs.umass.edu/kurose/transport/UDP.html

Quote

UDP, defined in [RFC 768],  does just about as little as a transport protocol can.   Aside from the multiplexing/demultiplexing function and some light error checking, it adds nothing to IP. In fact, if the application developer chooses UDP instead of TCP, then the application is talking almost directly with IP.

UDP takes messages from application process, attaches source and destination port number fields for the multiplexing/demultiplexing service, adds two other fields of minor importance, and passes the resulting "segment" to the network layer. The network layer encapsulates the segment into an IP datagram and then makes a best-effort attempt to deliver the segment to the receiving host.

If the segment arrives at the receiving host, UDP uses the port numbers and the IP source and destination addresses to deliver the data in the segment to the correct application process.


So the real question is what if any gaurnatees does IP give you.

Succint definiton of IP from  here:

http://www.freesoft.org/CIE/Topics/79.htm

Quote

IP is the Internet's most basic protocol. In order to function in a TCP/IP network, a network segment's only requirement is to forward IP packets. In fact, a TCP/IP network can be defined as a communication medium that can transport IP packets. Almost all other TCP/IP functions are constructed by layering atop IP.

IP is documented in RFC 791, and IP broadcasting procedures are discussed in RFC 919. The Encyclopedia's Programmed Instruction Course includes an IP Section.

IP is a datagram-oriented protocol, treating each packet independently. This means each packet must contain complete addressing information. Also, IP makes no attempt to determine if packets reach their destination or to take corrective action if they do not. Nor does IP checksum the contents of a packet, only the IP header.

IP provides several services:

   * Addressing. IP headers contain 32-bit addresses which identify the sending and receiving hosts. These addresses are used by intermediate routers to select a path through the network for the packet.

   * Fragmentation. IP packets may be split, or fragmented, into smaller packets. This permits a large packet to travel across a network which can only handle smaller packets. IP fragments and reassembles packets transparently.

   * Packet timeouts. Each IP packet contains a Time To Live (TTL) field, which is decremented every time a router handles the packet. If TTL reaches zero, the packet is discarded, preventing packets from running in circles forever and flooding a network.

   * Type of Service. IP supports traffic prioritization by allowing packets to be labeled with an abstract type of service.

   * Options. IP provides several optional features, allowing a packet's sender to set requirements on the path it takes through the network (source routing), trace the route a packet takes (record route), and label packets with security features.




So by my reading of these sources, which seem well researched with proper references back to the RFCs, I was correct and UDP indeed CAN both fragment and garbage data.


Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Offline endolf

JGO Coder


Medals: 7
Exp: 15 years


Current project release date: sometime in 3003


« Reply #7 - Posted 2005-04-18 04:41:50 »

Hi

I think the important bits to note are that fragmentation and reassembly can take place without you knowing, and there is no checksum on the data. The other parts of UDP (no reliability of packet arriving, no resending at protocol level etc) are known. This means that your data has a reasonable chance of not only not appearing at a remote side, but being a little on the screwy side. Just because your packet arrives, doesn't mean it's the same size or contents as the one you sent. Time to stick a checksum byte at the front of your packets peeps Smiley

Endolf

Offline kevglass

JGO Kernel


Medals: 186
Projects: 24
Exp: 18 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #8 - Posted 2005-04-18 05:30:38 »

While IP does fragment that doesn't effect you since according to that quote above:

Quote

IP fragments and reassembles packets transparently.


As to checksums, assuming ethernet (which of course we can't) you've got a CRC. I believe ATM the only other (protocol I know that used widely on the internet) also uses a CRC (sometimes referred to as a CFF).

The UDP header(http://www.networksorcery.com/enp/protocol/udp.htm) like the TCP header includes a checksum for the IP header, UDP header and data. Having written code to generate both the UDP and TCP checksums by hand in my previous job I'm scarred by this memory.

So, if you send a UDP packet at the other end you'll only recieve it if your IP packet has been reassembled and the data was correct (well as much as a 16bit checksum can give you).

Kev

Offline Jeff

JGO Coder




Got any cats?


« Reply #9 - Posted 2005-04-18 16:19:59 »

Quote
While IP does fragment that doesn't effect you since according to that quote above:


You are correct and I stand corrected.  
It was late and I missed that. Smiley  I do want to do more research though because I don't entirely believe it.  Im pretty sure, for instance, if you get over MTU, that you can see split packets.  If not, then you would never see splits in TCP because TCP is built ontop of UDP, but you do.



Quote

As to checksums, assuming ethernet (which of course we can't)


Yup we can't.  Thst the very definition of "internet", that you don't know the exact transports between you and the end.

Quote
I believe ATM the only other (protocol I know that used widely on the internet) also uses a CRC (sometimes referred to as a CFF).


Well when I was in college there was BitNet too.  I have no idea whats out there. If you want to make extra-protocol assumptiuosn  you can... but they may suddenly and inexplicably fail so I don't like to.


Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Jeff

JGO Coder




Got any cats?


« Reply #10 - Posted 2005-04-18 16:28:39 »

Whelp Kev, I give, you're right... here's a good high level explaination of reassembly:

Quote

# How a destination system reassembles the fragments of an IP datagram?

  1. When a host receives an IP fragment, it stores the fragment in a reassembly buffer based on its fragment offset field.
  2. Once all the fragments of the original IP datagram are received, the datagram is processed.
  3. Upon receiving the first fragment, a reassembly timer is started.
  4. If the reassembly timer expires before all the fragments are received, the datagram is discarded.


Nice IP fragementation FAQ  in general here:

http://www.geocities.com/SiliconValley/Vista/8672/network/ipfrag.html#A1

This raises the issue thopugh of why you should ever see partial packets out of a TCP stream.  The only reason I can think of is if bandwidth v. buffering of the stream on the sending end caused it to divide the data in transmission...





Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Offline William Denniss

JGO Coder


Projects: 2


Fire at will


« Reply #11 - Posted 2005-04-19 03:39:15 »

Indeed, that was also my understanding, you either read out your datagram in full at the other end (noting that if your datagram is larger than the ByteBuffer you read it into, the remaining bytes will be silently destroyed) or it never arrives.

I suspected this was the case when I started investigating UDP in NIO (really, if it wasn't the case, UDP would be a lot less useful).  However, I found is surprisingly hard to confirm.  I read several books, most which skim over UDP or just provide code samples and don't mention the fundimental concepts like this (Java Network Programming, 3rd Edition was pretty useless in this regard).

I did however find an excellent reference which confirmed my assumptions about UDP handling in NIO, the relevant section which I have quoted for you.

The book is the Java NIO book.

Quoting from section 3.5.4:
Quote
Invoking  send( ) sends the content of the given ByteBuffer object, from its current position to its limit, to the destination address and port described by the given SocketAddress object. If the DatagramChannel object is in blocking mode, the invoking thread may sleep until the datagram can be queued for transmission. If the channel is nonblocking, the return value will be either the number of bytes in the byte buffer or 0. Sending datagrams is an all-or-nothing proposition. If the transmit queue does not have sufficient room to hold the entire datagram, then nothing at all is sent.


Quote

Note that datagram protocols are inherently unreliable; they make no delivery guarantees. A nonzero return value from send( ) does not indicate that the datagram arrived at its destination, only that it was successfully queued to the local networking layer for transmission. Additionally, transport protocols along the way may fragment the datagram. Ethernet, for example, cannot transport packets larger than about 1,500 bytes. If your datagram is large, it runs the risk of being broken into pieces, multiplying the chances of packet loss in transit. The datagram will be reassembled at the destination, and the receiver won't see the fragments, but if any fragments fail to arrive in a timely manner, the entire datagram will be discarded.


If you have any NIO questions then I highly recommend that book!  I would have saved many hours of searching if that was the first book I found.

Cheers,

Will.

Offline Jeff

JGO Coder




Got any cats?


« Reply #12 - Posted 2005-04-19 03:59:19 »

Cool, I'll have to add it to my reference library.

Thanks Will!

JK

Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #13 - Posted 2005-04-19 04:47:43 »

Sorry for not replying earlier; very busy at the moment, and have a 1.5 hour conference lecture to give tomorrow (!).

Quote

I suspected this was the case when I started investigating UDP in NIO (really, if it wasn't the case, UDP would be a lot less useful).  However, I found is surprisingly hard to confirm.  I read several books


Did the TCP bible have nothing to say on fragmentation?

Quote

, most which skim over UDP or just provide code samples and don't mention the fundimental concepts like this (Java Network Programming, 3rd Edition was pretty useless in this regard).


Yeah, they're not worth the time of day. To a certain extent, it's legitimate that they don't bother, precisely because everyone has a copy of the bible Sad.

But, mainly, it's that the authors don't tend to know much about the details of their subject, AFAICS Sad.

Quote

If you have any NIO questions then I highly recommend that book!  I would have saved many hours of searching if that was the first book I found.


Note that the author apparently hasn't tried a lot of what he tells you to do (going by the fact that the book tells you to do things that for the first 2 years of publication were wrong or impossible). The book gives up at most points just as it's about to get into telling you something truly useful and difficult to deduce for yourself Sad. So ... YMMV. Personally, whilst i find it's not a bad book, I can't say it's particular good either - I was disappointed. Shrug.

malloc will be first against the wall when the revolution comes...
Offline dsellars

Junior Duke




Need to write more games


« Reply #14 - Posted 2005-04-19 06:13:35 »

I know this is a bit out of context, but I'm pretty sure that this isn't correct:

Quote
TCP is built ontop of UDP


The only thing they have in common it that they are both Transport protocols.

Dan.
Offline kevglass

JGO Kernel


Medals: 186
Projects: 24
Exp: 18 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #15 - Posted 2005-04-19 06:19:52 »

I'm sure he meant IP.

The segmentation you see at TCP isn't the same as the segmentation at IP. TCP segments are segments of the buffer that is being transmitted. When TCP sends out its segements via IP they could be resegmented but IP would be responsible for reassembling the segments before passing them up to TCP for it to perform it resegmentation Wink

Cooooool

Kev

Offline endolf

JGO Coder


Medals: 7
Exp: 15 years


Current project release date: sometime in 3003


« Reply #16 - Posted 2005-04-19 17:41:42 »

Did I miss the point of Kev's post when it comes to checksumming. I know that the header is checksummed. But what about the data?. If the packet gets fragmented, then it's either rebuild (fragment count wise) totally or not at all, but what about corruption during transmission?

There is the posibility of this, either part of a fragments data, or when not fragmented, the data of the packet might get corrupted, so how does this show that we don't need checksumming?.

Cheers

Endolf

Offline kevglass

JGO Kernel


Medals: 186
Projects: 24
Exp: 18 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #17 - Posted 2005-04-19 17:43:48 »

UDP performs a checksum on the DATA.

Kev

Offline endolf

JGO Coder


Medals: 7
Exp: 15 years


Current project release date: sometime in 3003


« Reply #18 - Posted 2005-04-19 17:45:46 »

Re read RFC 768, must have misread it as I thought it was just the header, but ....

Quote
Checksum is the 16-bit one's complement of the one's complement sum of a
pseudo header of information from the IP header, the UDP header, and the
data,  padded  with zero octets  at the end (if  necessary)  to  make  a
multiple of two octets.


Endolf

Offline William Denniss

JGO Coder


Projects: 2


Fire at will


« Reply #19 - Posted 2005-04-19 23:05:34 »

Quote
Sorry for not replying earlier; very busy at the moment, and have a 1.5 hour conference lecture to give tomorrow (!).

Did the TCP bible have nothing to say on fragmentation?


I don't have that book, do you highly recommend it?

Quote

Yeah, they're not worth the time of day. To a certain extent, it's legitimate that they don't bother, precisely because everyone has a copy of the bible Sad.

But, mainly, it's that the authors don't tend to know much about the details of their subject, AFAICS Sad.


I think that's exactly it, they didn't know much about it but figured they needed to have a chapter on it, so they just cooked up an example from the API without actually understanding it (stupid really, since understanding how it works is exactly why the reader is digging deeper and reading the book to begin with)

Quote

[re Java NIO book]

Note that the author apparently hasn't tried a lot of what he tells you to do (going by the fact that the book tells you to do things that for the first 2 years of publication were wrong or impossible). The book gives up at most points just as it's about to get into telling you something truly useful and difficult to deduce for yourself Sad. So ... YMMV. Personally, whilst i find it's not a bad book, I can't say it's particular good either - I was disappointed. Shrug.


You may be right.  I have found what I have read so far very useful, especially the paragraph's I quoted.  That's not to say it is the ultimate NIO reference, just that it answered beautifully a question I had searched quite hard to answer/confirm.  My comments on both books mentioned were purely related to their handling of DatagramChannel.

The section on Selectors in Java NIO also looks very good, I will soon be tackling that.

Cheers,

Will.

Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #20 - Posted 2005-04-21 09:12:30 »

LOL. At the ACCU conference, I read all of the security books. They're all crap.

One particularly memorable one has "world's best security reference" as part of the cover text. And yet, with a whole chapter on Samba, it "forgets" to mention the hashing scheme used by Samba - MD4, otherwise known as "crackable by any moron in approximately 3 hours, and soon to be crackable instantaneously, we're just waiting for the freely available kit to be completed".

This is quite apart from failing to explain the key elements of the SMB password system, and the main gotchas, DESPITE the fact that I know they are written in to the manpages!

So much crap out there in books. Sigh.

malloc will be first against the wall when the revolution comes...
Offline whome

Junior Duke




Carte Noir Java


« Reply #21 - Posted 2005-07-15 16:11:32 »

I have used two terminator methods in packets over NIO socket.
a) terminator byte
I've usually used 0x00 (NULL) byte at the end of packet. Then I just read until a terminator is found and handle the completed packet. You cannot send random binary data this way if it might contain null bytes within data. Text based packets usually are fine.

b) length terminated
send length of data at the start of each packet, then read given num of bytes and handle the completed packet. Handles any random binary data.

Writing back to non-blocking socket is quite hard as you need to handle all-or-nothing pattern. I've added inQueue and outQueue lists to a socket class and queue pending outgoing packets. Then I just take oldest one and write it until bytebuffer.available() returns false. I use Selector OP_WRITE to flag writable sockets. Soon as I've completed all outgoing packets I remove OP_WRITE and socket is kept in OP_READ mode.
Offline Alan_W

JGO Knight


Medals: 8
Projects: 3


Java tames rock!


« Reply #22 - Posted 2005-07-19 06:36:11 »

Indeed, that was also my understanding, you either read out your datagram in full at the other end (noting that if your datagram is larger than the ByteBuffer you read it into, the remaining bytes will be silently destroyed) or it never arrives..

That's good to know.  I've been wondering whether sizing my ByteBuffer to suit my packets, would leave me open to buffer overruns, if a spurious large 3rd party packet arrived.

Alan

Time flies like a bird. Fruit flies like a banana.
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Longarmx (50 views)
2014-10-17 03:59:02

Norakomi (39 views)
2014-10-16 15:22:06

Norakomi (31 views)
2014-10-16 15:20:20

lcass (36 views)
2014-10-15 16:18:58

TehJavaDev (66 views)
2014-10-14 00:39:48

TehJavaDev (65 views)
2014-10-14 00:35:47

TehJavaDev (56 views)
2014-10-14 00:32:37

BurntPizza (72 views)
2014-10-11 23:24:42

BurntPizza (44 views)
2014-10-11 23:10:45

BurntPizza (84 views)
2014-10-11 22:30:10
Understanding relations between setOrigin, setScale and setPosition in libGdx
by mbabuskov
2014-10-09 22:35:00

Definite guide to supporting multiple device resolutions on Android (2014)
by mbabuskov
2014-10-02 22:36:02

List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!