Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
on:
2011-08-15 12:21:54 » |
|
1 2 3 4 5 6 7 8 9 10 11
| String readFromFile(String strFile) { File file = new File(strFile); URI uri = file.toURI(); byte[] bytes = null; try{ bytes = java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(uri)); }catch(IOException e) { e.printStackTrace(); return "ERROR loading file "+strFile; } return new String(bytes); } |
This is Java 7 mind you. Since its new I tried this and a couple of other stuff too. This is the fastest. Can you do faster ? Note: A file read by this should be able to have "no" size restriction.
|
|
|
|
Eli Delventhal
« League of Dukes » JGO Kernel      Posts: 3575 Medals: 44
Game Engineer
|
 |
«
Reply #1 on:
2011-08-15 13:05:48 » |
|
I always do: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
| try { BufferedReader br = new BufferedReader(new FileReader(file)); StringBuffer str = new StringBuffer(); String line = br.readLine(); while (line != null) { str.append(line); str.append("\n"); line = br.readLine(); }
return str.toString(); } catch |
I'll bet you a zillion dollars my way is slower. And more complicated.  It's just the way I first learned it and have been doing so for years.
|
See my work:OTC Software<br /> Currently Working On:Secret project... I edit JGO in production, because I simply don't waste time writing bugs
|
|
|
Z-Man
Sr. Member   Posts: 270 Medals: 7
|
 |
«
Reply #2 on:
2011-08-15 16:21:45 » |
|
I'll bet you a zillion dollars my way is slower. And more complicated.  It's just the way I first learned it and have been doing so for years. I don't know if this speeds it up much but it's one less append call so it can't hurt can it? 1 2
| str.append(line + "\n"); line = br.readLine(); |
I always read each line of a file containing text into an ArrayList so that I can get to each line by itself if I need to. Although with your method Eli you can probably just parse the line you need out of the String created by str, seems like more work to me though. This is what I do: 1 2 3 4 5 6 7 8
| ArrayList<String> fileContents = new ArrayList<>(); BufferedReader br = new BufferedReader(new FileReader(file)); String line = br.readLine(); while(line != null) { fileContents.add(line); line = br.readLine(); } |
As for your method Cero, it's probably fast but I don't like it because it uses bytes and I have no idea how to work with them >_< That's my fault though so you can ignore my opinion on it >_>
|
"Anything that can possibly go wrong, does."
|
|
|
Games published by our own members! Go get 'em!
|
|
Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
Reply #3 on:
2011-08-15 16:24:20 » |
|
As for your method Cero, it's probably fast but I don't like it because it uses bytes and I have no idea how to work with them >_< That's my fault though so you can ignore my opinion on it >_>
well its only internally. using the method you give the path and it returns a string, using the bytes, but its still a normal string after that. 1
| return new String(bytes); |
Also: what about writing ? plaintext mostly, using FileWriter atm I think
|
|
|
|
Z-Man
Sr. Member   Posts: 270 Medals: 7
|
 |
«
Reply #4 on:
2011-08-15 16:27:27 » |
|
As for your method Cero, it's probably fast but I don't like it because it uses bytes and I have no idea how to work with them >_< That's my fault though so you can ignore my opinion on it >_>
well its only internally. using the method you give the path and it returns a string, using the bytes, but its still a normal string after that. 1
| return new String(bytes); |
Also: what about writing ? plaintext mostly, using FileWriter atm I think Oh... well I like it more now. I had no idea that you could just cast a byte[] to a String. It kind of makes sense though since a String is just a collection of bytes >_< Mind if I use your method? As for writing I just use a BufferedWriter I belive. No idea if it's better/worse then a FileWriter for writing plain-text. EDIT: The only thing about your method of reading is, what do you do if you need to read from a text file that is inside a Jar?
|
"Anything that can possibly go wrong, does."
|
|
|
Riven
« League of Dukes » JGO Kernel      Posts: 5871 Medals: 255
Hand over your head.
|
 |
«
Reply #5 on:
2011-08-15 16:28:49 » |
|
I'll bet you a zillion dollars my way is slower. And more complicated.  It's just the way I first learned it and have been doing so for years. I don't know if this speeds it up much but it's one less append call so it can't hurt can it? 1
| str.append(line + "\n"); |
Yay. That would totally waste endless CPU cycles.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings
|
|
|
Z-Man
Sr. Member   Posts: 270 Medals: 7
|
 |
«
Reply #6 on:
2011-08-15 16:29:38 » |
|
I'll bet you a zillion dollars my way is slower. And more complicated.  It's just the way I first learned it and have been doing so for years. I don't know if this speeds it up much but it's one less append call so it can't hurt can it? 1
| str.append(line + "\n"); |
Yay. That would totally waste endless CPU cycles. O_O that sounds bad...
|
"Anything that can possibly go wrong, does."
|
|
|
Riven
« League of Dukes » JGO Kernel      Posts: 5871 Medals: 255
Hand over your head.
|
 |
«
Reply #7 on:
2011-08-15 16:32:58 » |
|
StringBuilder has a growing char[] backing it.
A line of code like: s = "abc"+def+"ghi"+jkl; is turned into: s = new StringBuilder("abc").append(def).append("ghi").append(jkl).toString();
So your suggestion would turn Eli's code into:
StringBuilder str; ... str.append(new StringBuilder(line).append("\n").toString());
Which means at least two/three new objects per line of input. (the StringBuilder and the char[], which is grown by the append)
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings
|
|
|
Z-Man
Sr. Member   Posts: 270 Medals: 7
|
 |
«
Reply #8 on:
2011-08-15 16:38:11 » |
|
Wow... that is worse. My bad.
|
"Anything that can possibly go wrong, does."
|
|
|
Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
Reply #9 on:
2011-08-15 17:02:44 » |
|
Mind if I use your method?
You don't need permission for code this short and simple =P EDIT: The only thing about your method of reading is, what do you do if you need to read from a text file that is inside a Jar?
Well I never do that. I structure everything like a C++ Game: Many Folders which hold all the content and an exe file. I wold care about compression and or formats, I could just compress/decompress it myself, but who cares. Only if gets > 700MB =P Oh yea using String concatenation while using StringBuilder.append is kinda silly. Call append each time. Well I guess most people don't write their own Map Editor which they rarely have to write stuff to file.
|
|
|
|
Games published by our own members! Go get 'em!
|
|
Abuse
JGO Kernel      Posts: 1866 Medals: 5
falling into the abyss of reality
|
 |
«
Reply #10 on:
2011-08-15 19:04:49 » |
|
The only problem I can see with your method Cero, is that you are assuming the platform's default charset is the same as the charset used when writing your data file. ( %29]new String(byte[])) Explicitly specify "UTF-8" both when reading & writing your Strings, and you shouldn't have any compatibility problems.
|
|
|
|
|
Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
Reply #11 on:
2011-08-15 19:20:26 » |
|
The only problem I can see with your method Cero, is that you are assuming the platform's default charset is the same as the charset used when writing your data file. ( %29]new String(byte[])) Explicitly specify "UTF-8" both when reading & writing your Strings, and you shouldn't have any compatibility problems. oh yeah thanks, just new String(bytes, "UTF8") but when writing I do it like this: 1 2 3 4 5 6 7 8
| void writeToFile(String msg, String path) { try{ BufferedWriter bw = new BufferedWriter(new FileWriter(path)); bw.write(msg); bw.close(); }catch(Exception e) { e.printStackTrace(); } } |
Not sure how to do it then. Well its not like the user has to open files my code writes... savegames, maps... all read by my code maybe the console log
|
|
|
|
counterp
Full Member   Posts: 235 Medals: 11
|
 |
«
Reply #12 on:
2011-08-15 20:26:21 » |
|
Well, how big is your average file? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
| public static final String readFile(String file) throws IOException { BufferedInputStream in = new BufferedInputStream(new FileInputStream(file)); ByteBuffer buffer = new ByteBuffer(); byte[] buf = new byte[1024]; int len; while ((len = in.read(buf)) != -1) { buffer.put(buf, len); } in.close(); return new String(buffer.buffer, 0, buffer.write); }
class ByteBuffer {
public byte[] buffer = new byte[256];
public int write;
public void put(byte[] buf, int len) { ensure(len); System.arraycopy(buf, 0, buffer, write, len); write += len; }
private void ensure(int amt) { int req = write + amt; if (buffer.length <= req) { byte[] temp = new byte[req * 2]; System.arraycopy(buffer, 0, temp, 0, write); buffer = temp; } }
} |
when reading a 249 byte file it was faster, when reading a 10,240,000 byte file it was faster, when reading a 10,240 byte file it was faster just gonna assume it's always faster side-note: NIO is not any faster than IO
|
|
|
|
|
CaptainJester
JGO Neuromancer     Posts: 1138 Medals: 8
Make it work; make it better.
|
 |
«
Reply #13 on:
2011-08-15 22:11:46 » |
|
Method I learned from Matzon: 1 2 3 4 5 6 7 8 9 10 11 12
| private String readFile(String fileName) throws IOException { BufferedInputStream fin = new BufferedInputStream(new FileInputStream(fileName)); ByteArrayOutputStream bout = new ByteArrayOutputStream(); byte buffer[] = new byte[8192]; int read = fin.read(buffer); while(read != -1) { bout.write(buffer, 0, read); read = fin.read(buffer); } fin.close(); return new String(bout.toByteArray()); } |
Tested it on file sizes: 387, 3KB and 8.3MB The new way is faster by 2/3 on the biggest and smallest. Surprisingly my way is slightly faster(109511ns compared to 170413ns) on the middle size.
|
|
|
|
counterp
Full Member   Posts: 235 Medals: 11
|
 |
«
Reply #14 on:
2011-08-15 22:24:28 » |
|
Mines is a little faster  (maybe because it doesn't have the abstractions of ByteArrayOutputStream?) but they look very similar, I didn't know other people used this method, I just thought it up when reading this thread.
|
|
|
|
|
Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
Reply #15 on:
2011-08-15 22:59:30 » |
|
Well, how big is your average file? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
| public static final String readFile(String file) throws IOException { BufferedInputStream in = new BufferedInputStream(new FileInputStream(file)); ByteBuffer buffer = new ByteBuffer(); byte[] buf = new byte[1024]; int len; while ((len = in.read(buf)) != -1) { buffer.put(buf, len); } in.close(); return new String(buffer.buffer, 0, buffer.write); }
class ByteBuffer {
public byte[] buffer = new byte[256];
public int write;
public void put(byte[] buf, int len) { ensure(len); System.arraycopy(buf, 0, buffer, write, len); write += len; }
private void ensure(int amt) { int req = write + amt; if (buffer.length <= req) { byte[] temp = new byte[req * 2]; System.arraycopy(buffer, 0, temp, 0, write); buffer = temp; } }
} |
when reading a 249 byte file it was faster, when reading a 10,240,000 byte file it was faster, when reading a 10,240 byte file it was faster just gonna assume it's always faster side-note: NIO is not any faster than IO this is fast. average 5 times faster, great stuff
|
|
|
|
counterp
Full Member   Posts: 235 Medals: 11
|
 |
«
Reply #16 on:
2011-08-15 23:54:08 » |
|
depending on the size of files, if you are reading really large files you might want to up the buffer from 1024 (to a max of 8192, after that there's only minimal change as far as I can tell)
|
|
|
|
|
ReBirth
JGO Wizard     Posts: 1279 Medals: 19
|
 |
«
Reply #17 on:
2011-08-16 01:29:08 » |
|
Is the '\n' also included on the resulted String?
|
Follow me, your mastah, on TWITTAH!
|
|
|
counterp
Full Member   Posts: 235 Medals: 11
|
 |
«
Reply #18 on:
2011-08-16 01:30:32 » |
|
It reads the whole file as opposed to line by line, so yes
|
|
|
|
|
BoBear2681
Full Member   Posts: 238 Medals: 8
|
 |
«
Reply #19 on:
2011-08-16 07:36:35 » |
|
counterp's and CaptainJester's/Matzon's techniques will keep "Windows" newlines as "\r\n" in the resulting String, but Eli's and Z-Man's won't, so there is a subtle difference. Not sure about Cero's since I haven't made the move to Java 7.
|
|
|
|
|
Riven
« League of Dukes » JGO Kernel      Posts: 5871 Medals: 255
Hand over your head.
|
 |
«
Reply #20 on:
2011-08-16 07:37:48 » |
|
Eli's version will also append a '\n' after the last line.
If he would read/write/read/write/read/write, the file would get larger and larger.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings
|
|
|
Eli Delventhal
« League of Dukes » JGO Kernel      Posts: 3575 Medals: 44
Game Engineer
|
 |
«
Reply #21 on:
2011-08-16 13:33:57 » |
|
Yeah don't worry Z-Man, it's not at all obvious that concatenating strings will be so incredibly slow. But a good rule of thumb is that if you are ever worried about speed when string combining is involved, always use a StringBuffer. If speed doesn't matter, don't bother, because it's uglier (as you already pointed out). Eli's version will also append a '\n' after the last line.
If he would read/write/read/write/read/write, the file would get larger and larger.
Yeah, in cases where this is an issue I just drop the last character every time. But I've never worried much about string file speed because I only ever do it once and I never have massive files. But I did once have a case where I was reading in a file, editing it, and then re-saving it and I had to deal with the trailing \n.
|
See my work:OTC Software<br /> Currently Working On:Secret project... I edit JGO in production, because I simply don't waste time writing bugs
|
|
|
Addictman
Full Member   Posts: 204 Medals: 5
Java games rock!
|
 |
«
Reply #22 on:
2011-08-16 14:33:01 » |
|
Obviously this method has some theoretical limitations to size, but you could set some kind of size limit where if size > limit, choose the buffered approach. At least it's fast. 1 2 3 4 5 6 7 8
| public final static String readFile(String fileName) throws IOException { File f = new File(fileName); FileInputStream fstream = new FileInputStream(f); byte[] bytes = new byte[(int) f.length()]; fstream.read(bytes); fstream.close(); return new String(bytes); } |
|
|
|
|
|
counterp
Full Member   Posts: 235 Medals: 11
|
 |
«
Reply #23 on:
2011-08-16 22:50:27 » |
|
I hadn't thought this would be viable for small files, but it actually performs very well. And the list goes on and on  1 2 3 4 5 6 7
| public final static String readFile(String file) throws IOException { FileChannel channel = new FileInputStream(new File(file)).getChannel(); ByteBuffer buffer = ByteBuffer.allocate((int) channel.size()); channel.read(buffer); channel.close(); return new String(buffer.array()); } |
I don't think it gets much faster than this (when you're reading large files everythings' pretty close in speed)
|
|
|
|
|
Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
Reply #24 on:
2011-08-17 08:47:58 » |
|
I hadn't thought this would be viable for small files, but it actually performs very well. And the list goes on and on  1 2 3 4 5 6 7
| public final static String readFile(String file) throws IOException { FileChannel channel = new FileInputStream(new File(file)).getChannel(); ByteBuffer buffer = ByteBuffer.allocate((int) channel.size()); channel.read(buffer); channel.close(); return new String(buffer.array()); } |
I don't think it gets much faster than this (when you're reading large files everythings' pretty close in speed) using FileChannel ? nah, when I benchmarked that it was slow maybe not the slowest but slower than your previous code and the java7 method.
|
|
|
|
counterp
Full Member   Posts: 235 Medals: 11
|
 |
«
Reply #25 on:
2011-08-17 09:01:53 » |
|
interesting, using java 7 it comes up as slightly faster for larger files and noticeably faster for small files.
how are you doing your benchmark? make sure you're running each method in its own instance of the JVM
|
|
|
|
|
Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
Reply #26 on:
2011-08-17 11:31:15 » |
|
interesting, using java 7 it comes up as slightly faster for larger files and noticeably faster for small files.
how are you doing your benchmark? make sure you're running each method in its own instance of the JVM
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
| long before = System.nanoTime(); for (int i=0;i<100;i++) { Util.readFromFile(path); } System.out.println("1: "+((System.nanoTime()-before)/1000L)+"us"); before = System.nanoTime(); for (int i=0;i<100;i++) { Util.readFileWithChannel(path); } System.out.println("2: "+((System.nanoTime()-before)/1000L)+"us"); before = System.nanoTime(); for (int i=0;i<100;i++) { Util.readFromFileJava7(path); } System.out.println("3: "+((System.nanoTime()-before)/1000L)+"us"); |
1 2 3
| 1: 305061us 2: 1579026us 3: 1476928us |
file is 500kb in this run.
|
|
|
|
Riven
« League of Dukes » JGO Kernel      Posts: 5871 Medals: 255
Hand over your head.
|
 |
«
Reply #27 on:
2011-08-17 11:36:09 » |
|
You're making the classic mistake in file performance benchmarking: you forget that the OS will cache any file that was recently read.
For a realistic benchmark, overwrite the file prior to each time you read the file.
In the above benchmark, that means overwriting the file 300 times, otherwise your results are useless.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings
|
|
|
Cero
JGO Neuromancer     Posts: 1050 Medals: 18
|
 |
«
Reply #28 on:
2011-08-17 11:50:26 » |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
| int iterations = 20; long before = System.nanoTime(); scrambleFile(path);
for (int i=0;i<iterations;i++) { Util.readFromFileJava7(path); scrambleFile(path); } System.out.println("1: "+((System.nanoTime()-before)/1000L/1000L)+"ms"); before = System.nanoTime(); for (int i=0;i<iterations;i++) { Util.readFileWithChannel(path); scrambleFile(path); } System.out.println("2: "+((System.nanoTime()-before)/1000L/1000L)+"ms"); before = System.nanoTime(); for (int i=0;i<iterations;i++) { Util.readFromFile(path); scrambleFile(path); } System.out.println("3: "+((System.nanoTime()-before)/1000L/1000L)+"ms\n\n"); long r = 0L; for (int i=0;i<iterations;i++) { before = System.nanoTime(); scrambleFile(path); r += ((System.nanoTime()-before)/1000L/1000L); } System.out.println("Scramble test average: "+(r/(float)iterations)+"ms"); scrambleFile(path); |
1 2 3 4 5 6
| 1: 668ms 2: 614ms 3: 655ms
Scramble test average: 29.45ms |
Seems to faster now, although the differences are getting very small here
|
|
|
|
Riven
« League of Dukes » JGO Kernel      Posts: 5871 Medals: 255
Hand over your head.
|
 |
«
Reply #29 on:
2011-08-17 11:56:17 » |
|
This is what I wanted to point out: in the end the harddisk is limiting factor, the CPU is mostly idling, so the code is largely irrelevant, unless it's extremely inefficient.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings
|
|
|
|