Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (477)
Games in Android Showcase (106)
games submitted by our members
Games in WIP (533)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1] 2
  ignore  |  Print  
  Read File to String: can you do faster ?  (Read 20203 times)
0 Members and 1 Guest are viewing this topic.
Online Cero
« Posted 2011-08-15 18:21:54 »

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
String readFromFile(String strFile)
    {
       File file = new File(strFile);
       URI uri = file.toURI();
       byte[] bytes = null;
       try{
          bytes = java.nio.file.Files.readAllBytes(java.nio.file.Paths.get(uri));
       }catch(IOException e) { e.printStackTrace(); return "ERROR loading file "+strFile; }
   
       return new String(bytes);
    }


This is Java 7 mind you. Since its new I tried this and a couple of other stuff too.
This is the fastest.

Can you do faster ?

Note: A file read by this should be able to have "no" size restriction.

Offline Eli Delventhal

JGO Kernel


Medals: 42
Projects: 11
Exp: 10 years


Game Engineer


« Reply #1 - Posted 2011-08-15 19:05:48 »

I always do:

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
try
{
    BufferedReader br = new BufferedReader(new FileReader(file));
    StringBuffer str = new StringBuffer();
    String line = br.readLine();
    while (line != null)
    {
        str.append(line);
        str.append("\n");
        line = br.readLine();
    }

    return str.toString();
}
catch //etc.


I'll bet you a zillion dollars my way is slower. And more complicated. Tongue It's just the way I first learned it and have been doing so for years.

See my work:
OTC Software
Offline Z-Man
« Reply #2 - Posted 2011-08-15 22:21:45 »

I'll bet you a zillion dollars my way is slower. And more complicated. Tongue It's just the way I first learned it and have been doing so for years.
I don't know if this speeds it up much but it's one less append call so it can't hurt can it?
1  
2  
str.append(line + "\n");
line = br.readLine();


I always read each line of a file containing text into an ArrayList so that I can get to each line by itself if I need to. Although with your method Eli you can probably just parse the line you need out of the String created by str, seems like more work to me though. This is what I do:
1  
2  
3  
4  
5  
6  
7  
8  
ArrayList<String> fileContents = new ArrayList<>(); // Please excuse the Java 7 code here, I'm lazy
BufferedReader br = new BufferedReader(new FileReader(file));
String line = br.readLine();
while(line != null)
{
   fileContents.add(line);
   line = br.readLine();
}


As for your method Cero, it's probably fast but I don't like it because it uses bytes and I have no idea how to work with them >_< That's my fault though so you can ignore my opinion on it >_>
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Online Cero
« Reply #3 - Posted 2011-08-15 22:24:20 »

As for your method Cero, it's probably fast but I don't like it because it uses bytes and I have no idea how to work with them >_< That's my fault though so you can ignore my opinion on it >_>

well its only internally. using the method you give the path and it returns a string, using the bytes, but its still a normal string after that.

1  
return new String(bytes); 



Also: what about writing ? plaintext mostly, using FileWriter atm I think

Offline Z-Man
« Reply #4 - Posted 2011-08-15 22:27:27 »

As for your method Cero, it's probably fast but I don't like it because it uses bytes and I have no idea how to work with them >_< That's my fault though so you can ignore my opinion on it >_>

well its only internally. using the method you give the path and it returns a string, using the bytes, but its still a normal string after that.

1  
return new String(bytes); 



Also: what about writing ? plaintext mostly, using FileWriter atm I think
Oh... well I like it more now. I had no idea that you could just cast a byte[] to a String. It kind of makes sense though since a String is just a collection of bytes >_< Mind if I use your method?

As for writing I just use a BufferedWriter I belive. No idea if it's better/worse then a FileWriter for writing plain-text.

EDIT: The only thing about your method of reading is, what do you do if you need to read from a text file that is inside a Jar?
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 743
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #5 - Posted 2011-08-15 22:28:49 »

I'll bet you a zillion dollars my way is slower. And more complicated. Tongue It's just the way I first learned it and have been doing so for years.
I don't know if this speeds it up much but it's one less append call so it can't hurt can it?
1  
str.append(line + "\n");

Yay. That would totally waste endless CPU cycles.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Z-Man
« Reply #6 - Posted 2011-08-15 22:29:38 »

I'll bet you a zillion dollars my way is slower. And more complicated. Tongue It's just the way I first learned it and have been doing so for years.
I don't know if this speeds it up much but it's one less append call so it can't hurt can it?
1  
str.append(line + "\n");

Yay. That would totally waste endless CPU cycles.
O_O that sounds bad...
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 743
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #7 - Posted 2011-08-15 22:32:58 »

StringBuilder has a growing char[] backing it.

A line of code like:
s = "abc"+def+"ghi"+jkl;
is turned into:
s = new StringBuilder("abc").append(def).append("ghi").append(jkl).toString();

So your suggestion would turn Eli's code into:

StringBuilder str;
...
str.append(new StringBuilder(line).append("\n").toString());


Which means at least two/three new objects per line of input. (the StringBuilder and the char[], which is grown by the append)

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Z-Man
« Reply #8 - Posted 2011-08-15 22:38:11 »

Wow... that is worse. My bad.
Online Cero
« Reply #9 - Posted 2011-08-15 23:02:44 »

Mind if I use your method?
You don't need permission for code this short and simple =P


EDIT: The only thing about your method of reading is, what do you do if you need to read from a text file that is inside a Jar?
Well I never do that. I structure everything like a C++ Game: Many Folders which hold all the content and an exe file.
I wold care about compression and or formats, I could just compress/decompress it myself, but who cares. Only if gets > 700MB  =P


Oh yea using String concatenation while using StringBuilder.append is kinda silly. Call append each time.


Well I guess most people don't write their own Map Editor which they rarely have to write stuff to file.

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Abuse

JGO Coder


Medals: 11


falling into the abyss of reality


« Reply #10 - Posted 2011-08-16 01:04:49 »

The only problem I can see with your method Cero, is that you are assuming the platform's default charset is the same as the charset used when writing your data file. (%29]new String(byte[]))

Explicitly specify "UTF-8" both when reading & writing your Strings, and you shouldn't have any compatibility problems.

Make Elite IV:Dangerous happen! Pledge your backing at KICKSTARTER here! https://dl.dropbox.com/u/54785909/EliteIVsmaller.png
Online Cero
« Reply #11 - Posted 2011-08-16 01:20:26 »

The only problem I can see with your method Cero, is that you are assuming the platform's default charset is the same as the charset used when writing your data file. (%29]new String(byte[]))

Explicitly specify "UTF-8" both when reading & writing your Strings, and you shouldn't have any compatibility problems.

oh yeah thanks, just   new String(bytes, "UTF8")


but when writing I do it like this:
1  
2  
3  
4  
5  
6  
7  
8  
void writeToFile(String msg, String path)
    {
       try{
          BufferedWriter bw = new BufferedWriter(new FileWriter(path));
          bw.write(msg);
          bw.close();
       }catch(Exception e) { e.printStackTrace(); }
    }

Not sure how to do it then.
Well its not like the user has to open files my code writes... savegames, maps... all read by my code
maybe the console log

Offline counterp

Senior Member


Medals: 11



« Reply #12 - Posted 2011-08-16 02:26:21 »

Well, how big is your average file?

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
public static final String readFile(String file) throws IOException {
   BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
   ByteBuffer buffer = new ByteBuffer();
   byte[] buf = new byte[1024];
   int len;
   while ((len = in.read(buf)) != -1) {
      buffer.put(buf, len);
   }
   in.close();
   return new String(buffer.buffer, 0, buffer.write);
}

class ByteBuffer {

   public byte[] buffer = new byte[256];

   public int write;

   public void put(byte[] buf, int len) {
      ensure(len);
      System.arraycopy(buf, 0, buffer, write, len);
      write += len;
   }

   private void ensure(int amt) {
      int req = write + amt;
      if (buffer.length <= req) {
         byte[] temp = new byte[req * 2];
         System.arraycopy(buffer, 0, temp, 0, write);
         buffer = temp;
      }
   }

}


when reading a 249 byte file it was faster, when reading a 10,240,000 byte file it was faster, when reading a 10,240 byte file it was faster

just gonna assume it's always faster

side-note: NIO is not any faster than IO
Offline CaptainJester

JGO Knight


Medals: 12
Projects: 2
Exp: 14 years


Make it work; make it better.


« Reply #13 - Posted 2011-08-16 04:11:46 »

Method I learned from Matzon:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
private String readFile(String fileName) throws IOException {
    BufferedInputStream fin = new BufferedInputStream(new FileInputStream(fileName));
    ByteArrayOutputStream bout = new ByteArrayOutputStream();
    byte buffer[] = new byte[8192];
    int read = fin.read(buffer);
    while(read != -1) {
        bout.write(buffer, 0, read);
        read = fin.read(buffer);
    }
    fin.close();
    return new String(bout.toByteArray());
}

Tested it on file sizes: 387, 3KB and 8.3MB

The new way is faster by 2/3 on the biggest and smallest. Surprisingly my way is slightly faster(109511ns compared to 170413ns) on the middle size.

Offline counterp

Senior Member


Medals: 11



« Reply #14 - Posted 2011-08-16 04:24:28 »

Mines is a little faster Tongue (maybe because it doesn't have the abstractions of ByteArrayOutputStream?) but they look very similar, I didn't know other people used this method, I just thought it up when reading this thread.
Online Cero
« Reply #15 - Posted 2011-08-16 04:59:30 »

Well, how big is your average file?

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
public static final String readFile(String file) throws IOException {
   BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
   ByteBuffer buffer = new ByteBuffer();
   byte[] buf = new byte[1024];
   int len;
   while ((len = in.read(buf)) != -1) {
      buffer.put(buf, len);
   }
   in.close();
   return new String(buffer.buffer, 0, buffer.write);
}

class ByteBuffer {

   public byte[] buffer = new byte[256];

   public int write;

   public void put(byte[] buf, int len) {
      ensure(len);
      System.arraycopy(buf, 0, buffer, write, len);
      write += len;
   }

   private void ensure(int amt) {
      int req = write + amt;
      if (buffer.length <= req) {
         byte[] temp = new byte[req * 2];
         System.arraycopy(buffer, 0, temp, 0, write);
         buffer = temp;
      }
   }

}


when reading a 249 byte file it was faster, when reading a 10,240,000 byte file it was faster, when reading a 10,240 byte file it was faster

just gonna assume it's always faster

side-note: NIO is not any faster than IO

this is fast. average 5 times faster, great stuff

Offline counterp

Senior Member


Medals: 11



« Reply #16 - Posted 2011-08-16 05:54:08 »

depending on the size of files, if you are reading really large files you might want to up the buffer from 1024 (to a max of 8192, after that there's only minimal change as far as I can tell)
Offline ReBirth
« Reply #17 - Posted 2011-08-16 07:29:08 »

Is the '\n' also included on the resulted String?

Offline counterp

Senior Member


Medals: 11



« Reply #18 - Posted 2011-08-16 07:30:32 »

It reads the whole file as opposed to line by line, so yes
Offline BoBear2681

JGO Coder


Medals: 18



« Reply #19 - Posted 2011-08-16 13:36:35 »

counterp's and CaptainJester's/Matzon's techniques will keep "Windows" newlines as "\r\n" in the resulting String, but Eli's and Z-Man's won't, so there is a subtle difference.  Not sure about Cero's since I haven't made the move to Java 7.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 743
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #20 - Posted 2011-08-16 13:37:48 »

Eli's version will also append a '\n' after the last line.

If he would read/write/read/write/read/write, the file would get larger and larger.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Eli Delventhal

JGO Kernel


Medals: 42
Projects: 11
Exp: 10 years


Game Engineer


« Reply #21 - Posted 2011-08-16 19:33:57 »

Yeah don't worry Z-Man, it's not at all obvious that concatenating strings will be so incredibly slow. But a good rule of thumb is that if you are ever worried about speed when string combining is involved, always use a StringBuffer. If speed doesn't matter, don't bother, because it's uglier (as you already pointed out).

Eli's version will also append a '\n' after the last line.

If he would read/write/read/write/read/write, the file would get larger and larger.
Yeah, in cases where this is an issue I just drop the last character every time. But I've never worried much about string file speed because I only ever do it once and I never have massive files. But I did once have a case where I was reading in a file, editing it, and then re-saving it and I had to deal with the trailing \n.

See my work:
OTC Software
Offline Addictman

Senior Member


Medals: 3
Projects: 1


Java games rock!


« Reply #22 - Posted 2011-08-16 20:33:01 »

Obviously this method has some theoretical limitations to size, but you could set some kind of size limit where if size > limit, choose the buffered approach.

At least it's fast.

1  
2  
3  
4  
5  
6  
7  
8  
public final static String readFile(String fileName) throws IOException {
   File f = new File(fileName);
   FileInputStream fstream = new FileInputStream(f);
   byte[] bytes = new byte[(int) f.length()];
   fstream.read(bytes);
   fstream.close();
   return new String(bytes);
}
Offline counterp

Senior Member


Medals: 11



« Reply #23 - Posted 2011-08-17 04:50:27 »

I hadn't thought this would be viable for small files, but it actually performs very well. And the list goes on and on Wink

1  
2  
3  
4  
5  
6  
7  
public final static String readFile(String file) throws IOException {
   FileChannel channel = new FileInputStream(new File(file)).getChannel();
   ByteBuffer buffer = ByteBuffer.allocate((int) channel.size());
   channel.read(buffer);
   channel.close();
   return new String(buffer.array());
}


I don't think it gets much faster than this (when you're reading large files everythings' pretty close in speed)
Online Cero
« Reply #24 - Posted 2011-08-17 14:47:58 »

I hadn't thought this would be viable for small files, but it actually performs very well. And the list goes on and on Wink

1  
2  
3  
4  
5  
6  
7  
public final static String readFile(String file) throws IOException {
   FileChannel channel = new FileInputStream(new File(file)).getChannel();
   ByteBuffer buffer = ByteBuffer.allocate((int) channel.size());
   channel.read(buffer);
   channel.close();
   return new String(buffer.array());
}


I don't think it gets much faster than this (when you're reading large files everythings' pretty close in speed)

using FileChannel ? nah, when I benchmarked that it was slow
maybe not the slowest but slower than your previous code and the java7 method.

Offline counterp

Senior Member


Medals: 11



« Reply #25 - Posted 2011-08-17 15:01:53 »

interesting, using java 7 it comes up as slightly faster for larger files and noticeably faster for small files.

how are you doing your benchmark? make sure you're running each method in its own instance of the JVM
Online Cero
« Reply #26 - Posted 2011-08-17 17:31:15 »

interesting, using java 7 it comes up as slightly faster for larger files and noticeably faster for small files.

how are you doing your benchmark? make sure you're running each method in its own instance of the JVM
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
long before = System.nanoTime();
       
       for (int i=0;i<100;i++)
       {
          Util.readFromFile(path);
       }
       System.out.println("1: "+((System.nanoTime()-before)/1000L)+"us");
       
       
       before = System.nanoTime();
       for (int i=0;i<100;i++)
       {
          Util.readFileWithChannel(path);
       }
       System.out.println("2: "+((System.nanoTime()-before)/1000L)+"us");
       
       
       before = System.nanoTime();
       for (int i=0;i<100;i++)
       {
          Util.readFromFileJava7(path);
       }
       System.out.println("3: "+((System.nanoTime()-before)/1000L)+"us");


1  
2  
3  
1: 305061us
2: 1579026us
3: 1476928us


file is 500kb in this run.

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 743
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #27 - Posted 2011-08-17 17:36:09 »

You're making the classic mistake in file performance benchmarking: you forget that the OS will cache any file that was recently read.


For a realistic benchmark, overwrite the file prior to each time you read the file.

In the above benchmark, that means overwriting the file 300 times, otherwise your results are useless.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online Cero
« Reply #28 - Posted 2011-08-17 17:50:26 »

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
       int iterations = 20;
       long before = System.nanoTime();
       scrambleFile(path);

       for (int i=0;i<iterations;i++)
       {
          Util.readFromFileJava7(path);
          scrambleFile(path);
       }
       System.out.println("1: "+((System.nanoTime()-before)/1000L/1000L)+"ms");
       
       before = System.nanoTime();
       for (int i=0;i<iterations;i++)
       {
          Util.readFileWithChannel(path);
          scrambleFile(path);
       }
       System.out.println("2: "+((System.nanoTime()-before)/1000L/1000L)+"ms");
       
       before = System.nanoTime();
       for (int i=0;i<iterations;i++)
       {
          Util.readFromFile(path);
          scrambleFile(path);
       }
       System.out.println("3: "+((System.nanoTime()-before)/1000L/1000L)+"ms\n\n");
       
       long r = 0L;
       for (int i=0;i<iterations;i++)
       {
          before = System.nanoTime();
          scrambleFile(path);
          r += ((System.nanoTime()-before)/1000L/1000L);
       }
       System.out.println("Scramble test average: "+(r/(float)iterations)+"ms");
       scrambleFile(path);


1  
2  
3  
4  
5  
6  
1: 668ms
2: 614ms
3: 655ms


Scramble test average: 29.45ms


Seems to faster now, although the differences are getting very small here

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 743
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #29 - Posted 2011-08-17 17:56:17 »

This is what I wanted to point out: in the end the harddisk is limiting factor, the CPU is mostly idling, so the code is largely irrelevant, unless it's extremely inefficient.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Pages: [1] 2
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

pw (24 views)
2014-07-24 01:59:36

Riven (24 views)
2014-07-23 21:16:32

Riven (18 views)
2014-07-23 21:07:15

Riven (21 views)
2014-07-23 20:56:16

ctomni231 (50 views)
2014-07-18 06:55:21

Zero Volt (45 views)
2014-07-17 23:47:54

danieldean (36 views)
2014-07-17 23:41:23

MustardPeter (39 views)
2014-07-16 23:30:00

Cero (55 views)
2014-07-16 00:42:17

Riven (55 views)
2014-07-14 18:02:53
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!