Java-Gaming.org Hi !
Featured games (81)
games approved by the League of Dukes
Games in Showcase (513)
Games in Android Showcase (119)
games submitted by our members
Games in WIP (576)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  File transfer through a socket stream  (Read 2772 times)
0 Members and 1 Guest are viewing this topic.
Offline Endymius

Junior Newbie




100101001011


« Posted 2003-09-13 15:00:36 »

I'm trying to find the best possible way to transfer a file from the server to the client. I am hoping to use this method:

- Client sends request for a file ( in the Server dir )
- Server locates the file
- File is buffered and sent through a stream one section ( ~3000 bytes ) at a time
- Client reads the stream and writes the file

My question is, what is the best existing way to do the process of sending. Reader -> Stream (server) Stream -> Writer (client) ? ObjectInputStream and then a socket stream? I'm very confused about this issue and if someone would be kind enough to shed some light on it, I would be very thankful. Smiley
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #1 - Posted 2003-09-13 16:38:09 »

Quote
I'm trying to find the best possible way to transfer a file from the server to the client. I am hoping to use this method:

- Client sends request for a file ( in the Server dir )
- Server locates the file
- File is buffered and sent through a stream one section ( ~3000 bytes ) at a time
- Client reads the stream and writes the file

My question is, what is the best existing way to do the process of sending. Reader -> Stream (server) Stream -> Writer (client) ? ObjectInputStream and then a socket stream? I'm very confused about this issue and if someone would be kind enough to shed some light on it, I would be very thankful. Smiley


First questions you need to ask yourself (and answer for us Smiley) are:


  • What are your typical filesizes?
  • ...smallest?
  • ...biggest?
  • What's the context - why are you sending these files, what other processes are going on at the same time (is the server heavily loaded doing lots of CPU work, for instance?) - or is the server ONLY sending files?
  • When you say "what is the BEST way..." do you want the FASTEST, or the EASIEST to program, or the MOST EFFICIENT (i.e. takes smallest amount of CPU time etc)...?

malloc will be first against the wall when the revolution comes...
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #2 - Posted 2003-09-13 16:42:05 »

Quote

- File is buffered and sent through a stream one section ( ~3000 bytes ) at a time


3k is a strange number to choose (even roughly). 1k, 15K or 30K would make a lot of sense (each for different reasons - depends on what you are trying to do), so I'm wondering if there's any particular reason why you said roughly 3k ?

And several good ways of sending files in java you don't actually get to choose a number to send at once anyway....

malloc will be first against the wall when the revolution comes...
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Endymius

Junior Newbie




100101001011


« Reply #3 - Posted 2003-09-13 16:44:52 »

Allright Smiley

The typical file sizes can be of variable degree. Anything from 1kb to 5mb ( that's about a good range ).

There won't be any processes other than the file transfer happening at the same time, I don't see a need to thread the Server/ Client at this time.

By "best" I mean the best non error prone, logical, efficient way to send the files.

Also, i chose 3000 bytes out of my hat. I am not aware in what partitions transfer is most efficient that's why I'm asking Smiley
Offline Endymius

Junior Newbie




100101001011


« Reply #4 - Posted 2003-09-13 16:46:09 »

Quote


And several good ways of sending files in java you don't actually get to choose a number to send at once anyway....


Elaborate on this please Smiley
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #5 - Posted 2003-09-13 19:56:17 »

Quote
Allright Smiley

The typical file sizes can be of variable degree. Anything from 1kb to 5mb ( that's about a good range ).

There won't be any processes other than the file transfer happening at the same time, I don't see a need to thread the Server/ Client at this time.

By "best" I mean the best non error prone, logical, efficient way to send the files.


Without any special needs going on, I'd probably just do something like:

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
// Setup:
File f = new File( "/whereever/thingy" );
FileInputStream fis = new FileInputStream( f );
FileChannel rc = fis.getChannel();

...
// Loop:

int position = 0;
int fileLength = // length of file in bytes
if( position < fileLength )
{
rc.transferTo( position, fileLength-position, wbc );
}
else
{
out( "file transferred to nework Channel = "+wbc );
}


...where wbc is a SocketChannel returned from a ServerSocketChannel via:

1  
SocketChannel wbc = ((ServerSocketChannel) key.channel()).accept();


The loop section that writes the file to the wbc would be in the middle of the network-non-blocking select, thus:

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
while( true )
{
try
{
selector.select();
Set keys = selector.selectedKeys();
                       
                        Iterator i = keys.iterator();
                        while( i.hasNext() )
                        {
                              SelectionKey key = (SelectionKey) i.next();
                              i.remove();
                             
                              if( key.isWritable() )
                              {
                                    try
                                    {
                                          processWritableKey( key );
                                    }
                                    catch( IOException e )
                                    {
                                          key.cancel();
                                          closeChannel( key.channel() );
                                    }
                              }
                              else
                                    out( "Unexpected: key \""+key+"\" is not writable; this Selector should only contain Writable keys...." );
                        }
}


...and the "processWritableKey" method initiates the loop that writes to the file. Obviously, you have to keep track of state etc, but you only asked "what's the best way of doing this", so I'm assuming you're OK to fill in the blanks yourself.

That's a good combination of "logical" and "efficient". If you wanted ultra efficient, you could add bells and whistles, but with rapidly diminishing returns...If you don't lke NIO, or don't want to learn it, then you'll be back to methods that are inherently inefficient, and so you have to work harder to try and squeeze out performance.

Generally, I trust methods like "transferTo" to work - unless I have special needs in a given situation (like "this has GOT to be REALLY REALLY fast!!!" Wink). Quoting from the API docs:

    "This method is potentially much more efficient than a simple loop that reads from this channel and writes to the target channel. Many operating systems can transfer bytes directly from the filesystem cache to the target channel without actually copying them."

malloc will be first against the wall when the revolution comes...
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #6 - Posted 2003-09-13 20:04:49 »

Quote


Elaborate on this please Smiley


Well, when you're using non-blocking I/O, you should just say "send all of it" - and it'll come back and say "I actually sent 31,245 bytes". So you say "go back and finish it off, you lazy £%£ !". You don't get to choose. Note, however, that for REALLY high performance, when you want to start doing fancy tricks like variable QoS (google for it) - where you want to give some clients faster service than others - you can tell NIO "transfer NO MORE THAN X bytes". But it's optional, and effectively pointless in normal usage.

When you're using Buffered streams in 1.3.x and below, you often adopt a similar approach and just use "readLine()" which doesn't read any specific amount of data - it handles the buffering for you. Sometimes this is fine, sometimes it's stupid - all depends on circumstance. In theory, bearing in mind that you are already forsaking the higher performance of NIO, using the built-in buffering could be your best bet - because a JVM might have some clever tricks going on with doing native buffering. I wouldn't bet on it though.

OTOH, I've done basic "read from fileinputstream into byte[], write from byte[] to Socket.getOutputStream()" before with variable sizes of byte array and noticed little or no difference for changing sizes. That was probably an artifact of the buffering being done beneath the covers.

malloc will be first against the wall when the revolution comes...
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #7 - Posted 2003-09-13 20:12:48 »

Quote


3k is a strange number to choose (even roughly). 1k, 15K or 30K would make a lot of sense (each for different reasons - depends on what you are trying to do), so I'm wondering if there's any particular reason why you said roughly 3k ?

And several good ways of sending files in java you don't actually get to choose a number to send at once anyway....


1k == significantly less than the MTU of a standard TCP/IP link. So, you have a good chance that the entire 1k of data gets transmitted in a single packet. 1501 bytes could be really stupid, because it means that you are very likely to fragment your data into "one big packet, one tiny packet, one big packet, one tiny packet". Assuming the presence of a good OS and network-card, you'd hope that together they stuffed your data into a large buffer on the NIC, and only sent out consistently sized packets. However, there are an awful lot of utterly crap network cards (both real and virtual) lurking around...

15k == enough less than 16k that you have a resonably good chance of fitting inside the hardware send buffer of a typical  network card (although YMMV - I still use some cards with as low as 4k buffers).

30k == again, aiming to fit inside network-card buffers. This is for slightly better network cards - and also happens to be about right to fit inside the SEND BUFFER SIZE reported by the 1.4 JVM on most machines I use...but until I find a detailed description of precisely WHAT that buffer includes (is it something reported by the OS? by the NIC hardware driver? what?), I'd "Assume nothing, and just suck it and see...".

Those are just off the top of my head - I make no recommendations for any of them. Wait until you profile and find that you're not achieving as high a throughput as you'd expect... (unless you're using UDP, in which case how much data you send per packet is much more important - because the OS and the network card will NOT make any changes for you - so you should be planning it right from the start).

malloc will be first against the wall when the revolution comes...
Offline Endymius

Junior Newbie




100101001011


« Reply #8 - Posted 2003-09-13 20:16:05 »

wow... all I can say is I really appreciate the time and effort you have put into explaining all this Smiley
Offline Jeff

JGO Coder




Got any cats?


« Reply #9 - Posted 2003-10-28 07:59:41 »

Although I havent  looked at it in depth, NIO provides memory mapped files for just such types of applications.  Used properly they will allow you to route the data from file to socket within a modern OS and save major copying over-head.

Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #10 - Posted 2003-10-28 17:10:37 »

Quote
Although I havent  looked at it in depth, NIO provides memory mapped files for just such types of applications.  Used properly they will allow you to route the data from file to socket within a modern OS and save major copying over-head.


IIRC this is considerably slower than not mem-mapping the file, even on server-VM (where it's only a little a bit slower).

And if you're supposed to mem-map for this, then WTF is the

<code>FileChannel.transferTo(...)</code>

method for? I don't have access to javadocs at the mo (on a VERY slow dialup away from home), but IIRC that method is expressly for this purpose?

malloc will be first against the wall when the revolution comes...
Offline Jeff

JGO Coder




Got any cats?


« Reply #11 - Posted 2003-10-28 17:58:54 »

Your deepre into the API then I've gone.

What I *CAN* tell you is that this is preciosely what memory mapped files are for.  On an OS that supports proper memopry mapping it means that you never have to copy the data out of the system's buffer but can transfer it directly in the OS.

Likely the call you are askling about uses memory mapping, on Solaris at least.

JK

Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Longarmx (37 views)
2014-10-17 03:59:02

Norakomi (28 views)
2014-10-16 15:22:06

Norakomi (24 views)
2014-10-16 15:20:20

lcass (27 views)
2014-10-15 16:18:58

TehJavaDev (52 views)
2014-10-14 00:39:48

TehJavaDev (54 views)
2014-10-14 00:35:47

TehJavaDev (42 views)
2014-10-14 00:32:37

BurntPizza (64 views)
2014-10-11 23:24:42

BurntPizza (36 views)
2014-10-11 23:10:45

BurntPizza (77 views)
2014-10-11 22:30:10
Understanding relations between setOrigin, setScale and setPosition in libGdx
by mbabuskov
2014-10-09 22:35:00

Definite guide to supporting multiple device resolutions on Android (2014)
by mbabuskov
2014-10-02 22:36:02

List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!