Review my thread pattern?

[theagentd]

Hehe, this is getting kind of ridiculous. He's making a Minecraft server for god's sake. Single threaded IO should be better as it's easier to use.

Oh, was there an OP? 

Why hello 

Quite a conversation I started...haha

Your thread pattern looks good. However you should only multithread things that actually need it. If something executes really fast, the synchronization between the threads are going to make it slower. The best is probably to keep the number of threads dynamic. That way you can adapt the number of threads to the workload.

[nsigma]

What you're lacking is a benchmark relative to the argument we're having; that is, one comparing multithreaded vs singlethreaded writing

Yes, I've said this a couple of time.  And there's also a logical reason an OS or filesystem might treat these conditions differently (even if none do in actuality).

(either ncq or queue depth are the slowing factor, java blocks until the OS can queue IO requests. I'm going to take a wild guess and say nsigma doesn't have ncq (enabled))

Huh?  Well, the system reports ncq is enabled, and the queue depth as 31.

I'm intrigued from what I understand of ncq (given that my knowledge in this area could be counted on the fingers of no hands  ) why the results I reported above would suggest this - I would have thought the opposite.

[counterp]

unless your disk is fragmented, writing files sequentially like that should be faster without ncq (although probably the rest of your system would benefit from ncq). I guess this wasn't the main factor though

then my 2nd point applies, which is that the queue can't be processed in time when you continuously add requests that can't be completed in time making the queue fill up, causing writing/reading to slow at a constant rate as more things which need to be queued are slowly more delayed at being queued (as long as you are writing/reading at a constant, fast rate)

realistically, unless your running some popular website (i.e. not minecraft), you will not be writing/reading at this rate so this will never become a problem.

Games published by our own members! Go get 'em!

[Nate]

My fancy pants 256gb SSD (probably shaved some life off it :p):

wrote 1 files of 512MB in 0.819 sec, total throughput: 625.15265MB/sec
wrote 2 files of 512MB in 1.833 sec, total throughput: 558.64703MB/sec
wrote 3 files of 512MB in 2.488 sec, total throughput: 617.36334MB/sec
wrote 4 files of 512MB in 7.421 sec, total throughput: 275.9736MB/sec
wrote 5 files of 512MB in 10.72 sec, total throughput: 238.80597MB/sec
wrote 6 files of 512MB in 13.155 sec, total throughput: 233.52338MB/sec
wrote 7 files of 512MB in 15.02 sec, total throughput: 238.61517MB/sec
wrote 8 files of 512MB in 17.925 sec, total throughput: 228.50768MB/sec

The 3 files was consistently faster like the above.

Here's my 750gb HDD:

wrote 1 files of 512MB in 0.457 sec, total throughput: 1120.3501MB/sec
wrote 2 files of 512MB in 1.911 sec, total throughput: 535.8451MB/sec
wrote 3 files of 512MB in 2.506 sec, total throughput: 612.92896MB/sec
wrote 4 files of 512MB in 3.859 sec, total throughput: 530.70746MB/sec
wrote 5 files of 512MB in 41.242 sec, total throughput: 62.072643MB/sec

I got bored and stopped it. Then I ran it again:

wrote 1 files of 512MB in 13.607 sec, total throughput: 37.62769MB/sec
wrote 2 files of 512MB in 40.309 sec, total throughput: 25.403757MB/sec

Again I got bored and stopped it. Did you ruin my HDD controller?!

Yes. Obviously harddrives aren't meant to have data written to them. ... Isn't that just the OS caching the data in RAM and returning instantly?

Seems you have a bit of an attitude. SSDs have a limited number of program/erase cycles. The 750gb HDD should be faster than 37MB/s.

[theagentd]

My fancy pants 256gb SSD (probably shaved some life off it :p):

wrote 1 files of 512MB in 0.819 sec, total throughput: 625.15265MB/sec
wrote 2 files of 512MB in 1.833 sec, total throughput: 558.64703MB/sec
wrote 3 files of 512MB in 2.488 sec, total throughput: 617.36334MB/sec
wrote 4 files of 512MB in 7.421 sec, total throughput: 275.9736MB/sec
wrote 5 files of 512MB in 10.72 sec, total throughput: 238.80597MB/sec
wrote 6 files of 512MB in 13.155 sec, total throughput: 233.52338MB/sec
wrote 7 files of 512MB in 15.02 sec, total throughput: 238.61517MB/sec
wrote 8 files of 512MB in 17.925 sec, total throughput: 228.50768MB/sec

The 3 files was consistently faster like the above.

Here's my 750gb HDD:

wrote 1 files of 512MB in 0.457 sec, total throughput: 1120.3501MB/sec
wrote 2 files of 512MB in 1.911 sec, total throughput: 535.8451MB/sec
wrote 3 files of 512MB in 2.506 sec, total throughput: 612.92896MB/sec
wrote 4 files of 512MB in 3.859 sec, total throughput: 530.70746MB/sec
wrote 5 files of 512MB in 41.242 sec, total throughput: 62.072643MB/sec

I got bored and stopped it. Then I ran it again:

wrote 1 files of 512MB in 13.607 sec, total throughput: 37.62769MB/sec
wrote 2 files of 512MB in 40.309 sec, total throughput: 25.403757MB/sec

Again I got bored and stopped it. Did you ruin my HDD controller?!

Yes. Obviously harddrives aren't meant to have data written to them. ... Isn't that just the OS caching the data in RAM and returning instantly?

Seems you have a bit of an attitude. SSDs have a limited number of program/erase cycles. The 750gb HDD should be faster than 37MB/s.

I... I'm sorry... I didn't mean to insult you with my ingenious jokes... FYI I do own an SSD, and I'm full aware of the limited number of writes you can do.

[Nate]

Not insulted, just sarcasm out of the blue seemed odd.

[counterp]

I do believe counterp just got served.

And this is why you should avoid siding with people in situations where you have no knowledge of the topic concerned, especially when you have nothing to contribute.  

[theagentd]

Jesus, why are people so sensitive? And "related to the topic"? This thread went OT from page one. >_>

[nsigma]

then my 2nd point applies, which is that the queue can't be processed in time when you continuously add requests that can't be completed in time making the queue fill up, causing writing/reading to slow at a constant rate as more things which need to be queued are slowly more delayed at being queued (as long as you are writing/reading at a constant, fast rate)

Sorry  , still don't get what point you're trying to make here, or how it relates to the results I posted.  Are you saying this is the reason that the throughput for 1 file is slower because of this, or that this explains why I get consistent throughput no matter how many files are being written to?  It sounds more like an argument for throughput decreasing, though that may be how I'm reading what you've written.

I understand how queue saturation through memory use and ultimately drive write speed should be the limiting factor in a filesystem that can handle multiple concurrent writes effectively (which ext4 seems to be getting closer to).  Would I be right to suspect that all / most of those who have posted results to Riven's code have NCQ enabled, and therefore the relevant differences are queuing in the OS / filesystem and not the hardware?

I do believe counterp just got served.

And this is why you should avoid siding with people in situations where you have no knowledge of the topic concerned, especially when you have nothing to contribute. 

Quite! This was the post that made me want to join in this discussion, and read a bit more about the issues - as someone with no knowledge of the topic, I felt perfectly placed to jump in and side with no-one! 

[counterp]

then my 2nd point applies, which is that the queue can't be processed in time when you continuously add requests that can't be completed in time making the queue fill up, causing writing/reading to slow at a constant rate as more things which need to be queued are slowly more delayed at being queued (as long as you are writing/reading at a constant, fast rate)

Sorry   , still don't get what point you're trying to make here, or how it relates to the results I posted.  Are you saying this is the reason that the throughput for 1 file is slower because of this, or that this explains why I get consistent throughput no matter how many files are being written to?  It sounds more like an argument for throughput decreasing, though that may be how I'm reading what you've written.

I understand how queue saturation through memory use and ultimately drive write speed should be the limiting factor in a filesystem that can handle multiple concurrent writes effectively (which ext4 seems to be getting closer to).  Would I be right to suspect that all / most of those who have posted results to Riven's code have NCQ enabled, and therefore the relevant differences are queuing in the OS / filesystem and not the hardware?

Yes to your last point.
Did you try running my test by the way?

Games published by our own members! Go get 'em!

[ra4king]

It's fun taking sides,especially when I learn about this stuff after you guys explain it