Java-Gaming.org Hi !
Featured games (81)
games approved by the League of Dukes
Games in Showcase (513)
Games in Android Showcase (119)
games submitted by our members
Games in WIP (576)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: 1 [2] 3
  ignore  |  Print  
  Review my thread pattern?  (Read 9063 times)
0 Members and 1 Guest are viewing this topic.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #30 - Posted 2011-09-15 07:26:01 »

the actual speed comes from other things not directly related to writing files (for example, things like creating FileOutputStream objects can be done concurrently)
If you think the creation of objects is even relevant in file I/O, it really gets laughable.


i don't see why you keep holding on to the idea that when you multithread file writing, the disk is writing each file at the same time and therefore seeking a lot... that's not true. you only seek once per file
If you're reading/writing concurrently, you have to seek at every context-switch (thread switch).

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #31 - Posted 2011-09-15 07:26:19 »

To ensure 'worst case secenario', the 'concurrent' file I/O is done from a single thread. This is all the same for the harddisk, it just makes sure every write is done to a different file.

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
   public static void main(String[] args) throws IOException
   {
      int writeSize = 4 * 1024;
      int totalSize = 512 * 1024 * 1024;

      for (int fileCount = 1; fileCount <= 8; fileCount++)
      {
         FileOutputStream[] fos = new FileOutputStream[fileCount];
         for (int i = 0; i < fos.length; i++)
         {
            fos[i] = new FileOutputStream(new File("C:/test." + i + ".tmp"));
         }

         long t0 = System.currentTimeMillis();
         writeInterleavedStreams(fos, totalSize / writeSize, writeSize);
         long t1 = System.currentTimeMillis();

         int size = (totalSize / 1024 / 1024);
         int sec = (int) ((t1 - t0) / 1000);
         System.out.println("wrote " + fos.length + " files of " + size + "MB in " + sec + " sec, total throughput: " + (size / sec * fileCount) + "MB/sec");
      }
      System.out.println();
   }

   public static void writeInterleavedStreams(OutputStream[] streams, int writeCount, int writeSize) throws IOException
   {
      byte[] buf = new byte[writeSize];

      for (int i = 0; i < writeCount; i++)
      {
         for (OutputStream stream : streams)
         {
            stream.write(buf);
            stream.flush();
         }
      }

      for (OutputStream stream : streams)
      {
         stream.close();
      }
   }


1  
2  
3  
4  
5  
6  
7  
8  
wrote 1 files of 512MB in 3 sec, total throughput: 170MB/sec <--- bad-ass
wrote 2 files of 512MB in 32 sec, total throughput: 32MB/sec <--- once you concurrently write more than 1 file, it's all lost
wrote 3 files of 512MB in 45 sec, total throughput: 33MB/sec
wrote 4 files of 512MB in 78 sec, total throughput: 24MB/sec
wrote 5 files of 512MB in 116 sec, total throughput: 20MB/sec
wrote 6 files of 512MB in 166 sec, total throughput: 18MB/sec
wrote 7 files of 512MB in 182 sec, total throughput: 14MB/sec
wrote 8 files of 512MB in 224 sec, total throughput: 16MB/sec

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline ra4king

JGO Kernel


Medals: 350
Projects: 3
Exp: 5 years


I'm the King!


« Reply #32 - Posted 2011-09-15 08:23:24 »

I do believe counterp just got served. Cool

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Nate

JGO Kernel


Medals: 149
Projects: 4
Exp: 14 years


Esoteric Software


« Reply #33 - Posted 2011-09-15 10:21:18 »

My fancy pants 256gb SSD (probably shaved some life off it :p):
Quote
wrote 1 files of 512MB in 0.819 sec, total throughput: 625.15265MB/sec
wrote 2 files of 512MB in 1.833 sec, total throughput: 558.64703MB/sec
wrote 3 files of 512MB in 2.488 sec, total throughput: 617.36334MB/sec
wrote 4 files of 512MB in 7.421 sec, total throughput: 275.9736MB/sec
wrote 5 files of 512MB in 10.72 sec, total throughput: 238.80597MB/sec
wrote 6 files of 512MB in 13.155 sec, total throughput: 233.52338MB/sec
wrote 7 files of 512MB in 15.02 sec, total throughput: 238.61517MB/sec
wrote 8 files of 512MB in 17.925 sec, total throughput: 228.50768MB/sec
The 3 files was consistently faster like the above.

Here's my 750gb HDD:
Quote
wrote 1 files of 512MB in 0.457 sec, total throughput: 1120.3501MB/sec
wrote 2 files of 512MB in 1.911 sec, total throughput: 535.8451MB/sec
wrote 3 files of 512MB in 2.506 sec, total throughput: 612.92896MB/sec
wrote 4 files of 512MB in 3.859 sec, total throughput: 530.70746MB/sec
wrote 5 files of 512MB in 41.242 sec, total throughput: 62.072643MB/sec
I got bored and stopped it. Then I ran it again:
Quote
wrote 1 files of 512MB in 13.607 sec, total throughput: 37.62769MB/sec
wrote 2 files of 512MB in 40.309 sec, total throughput: 25.403757MB/sec
Again I got bored and stopped it. Did you ruin my HDD controller?!

Offline concerto49

Junior Duke





« Reply #34 - Posted 2011-09-15 11:33:15 »

Riven, try concurrently writing on the same file. It should be faster than writing serially on 1 file. It triggers NCQ. Ok, please test with NCQ enabled, i.e. AHCI. Also, Java's IO is written to work concurrently to some extent, so the said context switching won't occur if it's the same file.

High performance, fast network, affordable price VPS - Cloud Shards
Available in Texas, New York & Los Angeles
Need a VPS Upgrade?
Offline theagentd
« Reply #35 - Posted 2011-09-15 12:58:32 »

My fancy pants 256gb SSD (probably shaved some life off it :p):
Quote
wrote 1 files of 512MB in 0.819 sec, total throughput: 625.15265MB/sec
wrote 2 files of 512MB in 1.833 sec, total throughput: 558.64703MB/sec
wrote 3 files of 512MB in 2.488 sec, total throughput: 617.36334MB/sec
wrote 4 files of 512MB in 7.421 sec, total throughput: 275.9736MB/sec
wrote 5 files of 512MB in 10.72 sec, total throughput: 238.80597MB/sec
wrote 6 files of 512MB in 13.155 sec, total throughput: 233.52338MB/sec
wrote 7 files of 512MB in 15.02 sec, total throughput: 238.61517MB/sec
wrote 8 files of 512MB in 17.925 sec, total throughput: 228.50768MB/sec
The 3 files was consistently faster like the above.

Here's my 750gb HDD:
Quote
wrote 1 files of 512MB in 0.457 sec, total throughput: 1120.3501MB/sec
wrote 2 files of 512MB in 1.911 sec, total throughput: 535.8451MB/sec
wrote 3 files of 512MB in 2.506 sec, total throughput: 612.92896MB/sec
wrote 4 files of 512MB in 3.859 sec, total throughput: 530.70746MB/sec
wrote 5 files of 512MB in 41.242 sec, total throughput: 62.072643MB/sec
I got bored and stopped it. Then I ran it again:
Quote
wrote 1 files of 512MB in 13.607 sec, total throughput: 37.62769MB/sec
wrote 2 files of 512MB in 40.309 sec, total throughput: 25.403757MB/sec
Again I got bored and stopped it. Did you ruin my HDD controller?!
Yes. Obviously harddrives aren't meant to have data written to them. ... Isn't that just the OS caching the data in RAM and returning instantly?

Myomyomyo.
Offline nsigma
« Reply #36 - Posted 2011-09-15 13:17:02 »

Sorry, I feel the need to sit on the fence yet again - a PITA for you guys and for me!  Grin

Isn't what Riven and counterp are saying both right?

Riven is right about the inherently serial nature of the disk write, but as others have pointed out, in most modern OS's / file systems isn't the FileOutputStream actually writing to RAM to be later flushed to disk by the filesystem, and therefore the implication that each thread context shift results in a disk seek is not correct???

Oh, and the effect of the above will also be that benchmarks will be quite different across different file systems.

Praxis LIVE - open-source intermedia toolkit and live interactive visual editor
Digital Prisoners - interactive spaces and projections
Offline theagentd
« Reply #37 - Posted 2011-09-15 13:51:40 »

Hehe, this is getting kind of ridiculous. He's making a Minecraft server for god's sake. Single threaded IO should be better as it's easier to use.

Myomyomyo.
Offline nsigma
« Reply #38 - Posted 2011-09-15 13:53:27 »

Hehe, this is getting kind of ridiculous. He's making a Minecraft server for god's sake. Single threaded IO should be better as it's easier to use.

Oh, was there an OP?  Smiley

Praxis LIVE - open-source intermedia toolkit and live interactive visual editor
Digital Prisoners - interactive spaces and projections
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #39 - Posted 2011-09-15 14:13:12 »

in most modern OS's / file systems isn't the FileOutputStream actually writing to RAM to be later flushed to disk by the filesystem, and therefore the implication that each thread context shift results in a disk seek is not correct???
No, the RAM will be sync-ed with the storage device, in a blocking operation.

This applies for:
- FileOutputStream.flush()
- RandomAccessFile.getFD().sync();
- FileChannel.force(boolean)


If we didn't have such a guarantee, lots of critical applications (like databases) couldn't restore to a 'known state' after a crash.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline nsigma
« Reply #40 - Posted 2011-09-15 14:20:32 »

in most modern OS's / file systems isn't the FileOutputStream actually writing to RAM to be later flushed to disk by the filesystem, and therefore the implication that each thread context shift results in a disk seek is not correct???
No, the RAM will be sync-ed with the storage device, in a blocking operation.

This applies for:
- FileOutputStream.flush()
- RandomAccessFile.getFD().sync();
- FileChannel.force(boolean)


If we didn't have such a guarantee, lots of critical applications (like databases) couldn't restore to a 'known state' after a crash.


hmm ... from the JavaDoc for flush()
Quote
If the intended destination of this stream is an abstraction provided by the underlying operating system, for example a file, then flushing the stream guarantees only that bytes previously written to the stream are passed to the operating system for writing; it does not guarantee that they are actually written to a physical device such as a disk drive.

Praxis LIVE - open-source intermedia toolkit and live interactive visual editor
Digital Prisoners - interactive spaces and projections
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #41 - Posted 2011-09-15 14:38:04 »

1  
2  
3  
4  
5  
6  
7  
         for (OutputStream stream : streams)
         {
            stream.write(buf);
            stream.flush();
            //((FileOutputStream) stream).getFD().sync();
            //((FileOutputStream) stream).getChannel().force(false);
         }


It seems you are right flush() does not block, while sync() and force(...) do.


Still, you can observe the dramatic performance degradation when doing concurrent file writes on HDDs.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #42 - Posted 2011-09-15 14:41:06 »

Again I got bored and stopped it. Did you ruin my HDD controller?!
It seems concurrently writing to files results in incredible fragmentation (making it hard to find new free clusters), which somehow remains the case after deletion of the files.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline nsigma
« Reply #43 - Posted 2011-09-15 15:11:20 »

It seems you are right flush() does not block, while sync() and force(...) do.
But you wouldn't do these for every block of data you write.  To use your example above, there are databases which don't even bother to do this after every write.  There are other ways of achieving data integrity, and for that matter these still don't guarantee the data is actually written to the disk depending on the hardware.
Still, you can observe the dramatic performance degradation when doing concurrent file writes on HDDs.
I wasn't disputing you were seeing this, just your comment about head-seek for every context-shift.  It'd be interesting to see how this differs across filesystems and OS's.  For that matter, I assume you've tried the above benchmark with multiple threads as opposed to just multiple writes?  I could understand why these two circumstances could be treated very differently in the underlying filesystem.

Praxis LIVE - open-source intermedia toolkit and live interactive visual editor
Digital Prisoners - interactive spaces and projections
Offline sproingie

JGO Kernel


Medals: 202



« Reply #44 - Posted 2011-09-15 16:07:48 »

I don't think anyone disputes that concurrent access harms throughput.  The reasons why this are more likely to do with context switching overhead and buffer cache busting than any universal truths about physical properties of the HDD, since you can see the effect even on a SSD (though it seems to take more threads to do it).

Even putting aside I/O, there's plenty of other reasons to avoid using threads unless you really need parallel execution on multiple cores (and most I/O doesn't need it).  Even then you should at least be using java.util.concurrent.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #45 - Posted 2011-09-15 16:21:05 »

The reasons why this are more likely to do with context switching overhead and buffer cache busting than any universal truths about physical properties of the HDD
Roll Eyes The benchmark runs in a single thread and only uses a single 4K byte[].

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline nsigma
« Reply #46 - Posted 2011-09-15 16:57:22 »

I don't think anyone disputes that concurrent access harms throughput. 

Yes, I do!  If you'd said "I don't think anyone disputes that concurrent access harms throughput on some systems" maybe.  I've no doubt of the results people have seen so far, but the whole thing is very dependent on the underlying OS and filesystem.

Eg. Riven's benchmark on Linux with ext4

wrote 1 files of 512MB in 9 sec, total throughput: 56MB/sec
wrote 2 files of 512MB in 16 sec, total throughput: 64MB/sec
wrote 3 files of 512MB in 29 sec, total throughput: 51MB/sec
wrote 4 files of 512MB in 39 sec, total throughput: 52MB/sec
wrote 5 files of 512MB in 48 sec, total throughput: 50MB/sec
wrote 6 files of 512MB in 57 sec, total throughput: 48MB/sec
wrote 7 files of 512MB in 69 sec, total throughput: 49MB/sec
wrote 8 files of 512MB in 78 sec, total throughput: 48MB/sec


That seems fairly consistent to me.  Though my laptop HD is so slow to start with, I'm not sure it's a fair benchmark!  Grin

Even putting aside I/O, there's plenty of other reasons to avoid using threads unless you really need parallel execution on multiple cores (and most I/O doesn't need it).  Even then you should at least be using java.util.concurrent.

Not sure exactly what you're getting at here, but there are plenty of good reasons to use Threads.  While I/O is not necessarily one of them, I'm not sure unnecessarily serializing (synchronizing) your threading model for I/O is always justified either.

 ... just off to delete all them tmp files before I forget they're there and wonder why my disk has shrunk.  Roll Eyes

Praxis LIVE - open-source intermedia toolkit and live interactive visual editor
Digital Prisoners - interactive spaces and projections
Offline sproingie

JGO Kernel


Medals: 202



« Reply #47 - Posted 2011-09-15 16:58:03 »

Sounds like you're measuring burst latency then.  The valuable lesson we learned in that case how badly HDD makers lie. Smiley
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #48 - Posted 2011-09-15 17:01:13 »

Eg. Riven's benchmark on Linux with ext4
Interesting stuff, didn't realize it could be simply NTFS being crappy at concurrent access.

Then again, maybe EXT4 is just always slow Smiley

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #49 - Posted 2011-09-15 17:03:09 »

Sounds like you're measuring burst latency then.  The valuable lesson we learned in that case how badly HDD makers lie. Smiley

Can we assume FileOutputStream.close() sync's to the storage device?

If so, then my benchmark still stands persecutioncomplex

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline sproingie

JGO Kernel


Medals: 202



« Reply #50 - Posted 2011-09-15 17:06:32 »

ext4 can be pretty zippy but it depends on how you tune it.  Out of the box settings tend to be paranoid and use synchronous journaling, and on a laptop it's probably a good idea to keep it that way.

Talking to a guy now who's working on a supercomputer with 6TB of RAM.  I could probably avoid hitting the disk on that baby Smiley
Offline theagentd
« Reply #51 - Posted 2011-09-15 17:35:04 »

Sounds like you're measuring burst latency then.  The valuable lesson we learned in that case how badly HDD makers lie. Smiley

Can we assume FileOutputStream.close() sync's to the storage device?

If so, then my benchmark still stands persecutioncomplex
I wouldn't assume that. I don't think Java would force something like that, as it's an OS feature. Most of the time with hard drive access you aren't interested in the data actually being written to the hard drive immediately, only that it will be written eventually. If close() forced a cache flush, it would negate everything gained from actually having the cache in the first place.

Myomyomyo.
Offline Eli Delventhal

JGO Kernel


Medals: 42
Projects: 11
Exp: 10 years


Game Engineer


« Reply #52 - Posted 2011-09-15 17:38:31 »

Here's the results from running this on an HFS journaled (Mac OS X) laptop.

1  
2  
3  
4  
5  
6  
7  
8  
wrote 1 files of 512MB in 9 sec, total throughput: 56MB/sec
wrote 2 files of 512MB in 31 sec, total throughput: 32MB/sec
wrote 3 files of 512MB in 36 sec, total throughput: 42MB/sec
wrote 4 files of 512MB in 49 sec, total throughput: 40MB/sec
wrote 5 files of 512MB in 64 sec, total throughput: 40MB/sec
wrote 6 files of 512MB in 97 sec, total throughput: 30MB/sec
wrote 7 files of 512MB in 146 sec, total throughput: 21MB/sec
wrote 8 files of 512MB in 150 sec, total throughput: 24MB/sec

See my work:
OTC Software
Offline nsigma
« Reply #53 - Posted 2011-09-15 17:59:49 »

Eg. Riven's benchmark on Linux with ext4
Interesting stuff, didn't realize it could be simply NTFS being crappy at concurrent access.

Well, I did mention modern OS's earlier  Tongue

Then again, maybe EXT4 is just always slow Smiley

er .. yeah, modern OS .. consistently slow!  Smiley

Can we assume FileOutputStream.close() sync's to the storage device?
I wouldn't assume that. I don't think Java would force something like that, as it's an OS feature. ... If close() forced a cache flush, it would negate everything gained from actually having the cache in the first place.

I agree.  In fact, I'd go as far as to say it's safe to assume it doesn't.  I've found something definitive on Android, but not on Java yet.  However, I assume that's the point in having the FD.sync() method in the first place.  Maybe add that into the benchmark?  And threads too - interested to know if there's any threadlocal stuff affecting caching (I'm too busy lazy to write myself atm).

@Eli - phew, not just me that slow then - I was getting disk envy  persecutioncomplex

Praxis LIVE - open-source intermedia toolkit and live interactive visual editor
Digital Prisoners - interactive spaces and projections
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #54 - Posted 2011-09-15 18:02:14 »

I think it's pretty easy to rule out the OS caching data: writing so much that the OS simply can't cache it anymore.

If you write a file that is roughly 4 times the available RAM, and it has the same performance as writing a file about as big as the available RAM, you pretty much know (almost) everything is written to disk when the stream is closed (within a small margin of error).

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
   public static void main(String[] args) throws IOException
   {
      int writeSize = 4 * 1024;
      long minTotalSize = 1L * 1024 * 1024 * 1024;
      long maxTotalSize = 8L * 1024 * 1024 * 1024;

      int p = 0;

      for (long totalSize = minTotalSize; totalSize <= maxTotalSize; totalSize *= 2)
      {
         int fileCount = 1;
         FileOutputStream[] fos = new FileOutputStream[fileCount];
         for (int i = 0; i < fos.length; i++)
         {
            File file = new File("C:/test." + (++p) + ".tmp");
            file.delete();
            fos[i] = new FileOutputStream(file);
         }

         long t0 = System.currentTimeMillis();
         writeInterleavedStreams(fos, (int) (totalSize / writeSize), writeSize);
         long t1 = System.currentTimeMillis();

         long size = (totalSize / 1024 / 1024);
         long sec = ((t1 - t0) / 1000);
         System.out.println("wrote " + fos.length + " files of " + size + "MB in " + sec + " sec, total throughput: " + (size / sec * fileCount) + "MB/sec");
      }
   }

   public static void writeInterleavedStreams(OutputStream[] streams, int writeCount, int writeSize) throws IOException
   {
      byte[] buf = new byte[writeSize];

      for (int i = 0; i < writeCount; i++)
      {
         for (OutputStream stream : streams)
         {
            stream.write(buf);
         }
      }

      for (OutputStream stream : streams)
      {
         stream.close();
      }
   }


2GB RAM free, on a 4GB system:
1  
2  
3  
4  
wrote 1 files of 1024MB in 6 sec, total throughput: 170MB/sec
wrote 1 files of 2048MB in 13 sec, total throughput: 157MB/sec
wrote 1 files of 4096MB in 24 sec, total throughput: 170MB/sec
wrote 1 files of 8192MB in 50 sec, total throughput: 163MB/sec

That pretty much rules out OS caching (affecting the benchmark) to me.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline sproingie

JGO Kernel


Medals: 202



« Reply #55 - Posted 2011-09-15 20:58:36 »

If you want to force fsync, you have to call the outputstream's getFD().sync() .  Closing a stream only flushes it, which means the data is no longer the application's problem, but the OS and filesystem is free to buffer it as long as it feels like.

fsync() itself is only a very strong suggestion though, and HDD manufacturers often still cache things on the onboard RAM on the drive, which won't survive a power failure. 


Offline Riven
« League of Dukes »

JGO Overlord


Medals: 816
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #56 - Posted 2011-09-15 21:04:47 »

At some point you have to ensure the data is written to persistent storage, like prior to system shutdown / reboot. Same applies for 'safe removal' of external devices. Obviously this is done at the OS level, but I'd highly doubt we can't access that functionality somehow (without actually shutting down the device).

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline sproingie

JGO Kernel


Medals: 202



« Reply #57 - Posted 2011-09-15 21:12:32 »

You can force fsync on individual fd's, but I don't think java exposes the sync syscall.  Other than calling Runtime.getRuntime.exec("/bin/sync") and being SOL on Windows, that is...
Offline counterp

Senior Duke


Medals: 11



« Reply #58 - Posted 2011-09-16 00:22:07 »

Riven your benchmark is nice and all...... but writing files sequentially at that size and amount on the same thread will also quickly slow down the harddrive at the SAME RATE.... I don't get what point your trying to make..

It doesn't change the fact that single threaded is still slower than multi threaded lol...

You're also forgetting that 1) max file size is 2MB, and 2) minecraft will not be saving these files that often (you would have to be writing thousands at the same time or sequentially to get that kind of slowdown)

And I said object creation was only an example of one of the factors that can be done concurrently (parallel) on different threads. (creating a FileOutputStream object is actually really slow, almost an entire millisecond, although most of that is probably the actual opening of the file, there are still bulky operations like checking permissions)

What you're lacking is a benchmark relative to the argument we're having; that is, one comparing multithreaded vs singlethreaded writing (which would clearly prove I'm right, I've posted one, you should post your own too, just to make sure that my results aren't biased or anything)

(either ncq or queue depth are the slowing factor, java blocks until the OS can queue IO requests. I'm going to take a wild guess and say nsigma doesn't have ncq (enabled))

EDIT: consider this

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
   public static void main(String[] args) throws IOException {
      long start;
      int i = 0;
      long avg = 0;
      FileOutputStream out = new FileOutputStream(new File("large"));
      while (true) {
         start = System.nanoTime();

         out.write(new byte[1024 * 1024]);
         out.flush();

         long now = System.nanoTime();
         avg += (now - start) / 1000000;
         System.out.println("Average delay = " + (avg / ++i));
      }
   }


make sure you delete the large file (in your root project directory) afterwards, you should stop the program after you see the delay go up, if you wanna see throughput it's only a small modification to the above
Offline Conner_

Senior Newbie




~ Java Game Dev ~


« Reply #59 - Posted 2011-09-16 02:30:15 »

Hehe, this is getting kind of ridiculous. He's making a Minecraft server for god's sake. Single threaded IO should be better as it's easier to use.

Oh, was there an OP?  Smiley

Why hello  Tongue

Quite a conversation I started...haha

Find me on GitHub: connergdavis
My current project is called Obsidian Framework and is a private Minecraft server application used to play Multiplayer.
Pages: 1 [2] 3
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Longarmx (38 views)
2014-10-17 03:59:02

Norakomi (28 views)
2014-10-16 15:22:06

Norakomi (24 views)
2014-10-16 15:20:20

lcass (28 views)
2014-10-15 16:18:58

TehJavaDev (53 views)
2014-10-14 00:39:48

TehJavaDev (54 views)
2014-10-14 00:35:47

TehJavaDev (42 views)
2014-10-14 00:32:37

BurntPizza (64 views)
2014-10-11 23:24:42

BurntPizza (36 views)
2014-10-11 23:10:45

BurntPizza (78 views)
2014-10-11 22:30:10
Understanding relations between setOrigin, setScale and setPosition in libGdx
by mbabuskov
2014-10-09 22:35:00

Definite guide to supporting multiple device resolutions on Android (2014)
by mbabuskov
2014-10-02 22:36:02

List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!