Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (477)
Games in Android Showcase (107)
games submitted by our members
Games in WIP (536)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Traverse huge directories instantly (iteration like in JDK7, now!)  (Read 2607 times)
0 Members and 1 Guest are viewing this topic.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 748
Projects: 4
Exp: 16 years


Hand over your head.


« Posted 2009-12-01 16:21:00 »

This was so trivial, I wonder why Sun didn't do it in the first place...

It launches a new process and listens for the stdout:
 - Windows: cmd /c dir /B $path
 - Linux/Unix: /bin/ls $path

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
69  
70  
71  
72  
73  
74  
75  
76  
77  
78  
79  
80  
81  
82  
83  
84  
85  
86  
87  
88  
89  
public interface DirectoryVisitor
{
   public void visit(File parent, String name);
}

public class DirectoryIterator
{
   public static void main(String[] args) throws Exception
   {
      DirectoryVisitor visitor = new DirectoryVisitor()
      {
         @Override
         public void visit(File parent, String name)
         {
            System.out.println("file: " + name);
         }
      };

      iterate(new File(args[0]), visitor);
   }

   public static void iterate(File directory, DirectoryVisitor visitor)
   {
      String path = directory.getAbsolutePath();

      if (!directory.isDirectory())
      {
         throw new IllegalStateException("file not a directory: " + path);
      }

      if (System.getProperty("os.name").toLowerCase().contains("windows"))
      {
         try
         {
            String[] cmd = new String[5];
            cmd[0] = "cmd";
            cmd[1] = "/c";
            cmd[2] = "dir";
            cmd[3] = "/B"; // by: irreversible_kev
           cmd[4] = path;
            Process p = Runtime.getRuntime().exec(cmd);
            BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
            while (true)
            {
               String line = br.readLine();
               if (line == null)
                  break;
               visitor.visit(directory, line);
            }
            int ev = p.waitFor();
            if (ev != 0)
            {
               throw new IllegalStateException("cmd exit value: " + ev);
            }
         }
         catch (Exception exc)
         {
            throw new IllegalStateException(exc);
         }
      }
      else
      {
         try
         {
            String[] cmd = new String[2];
            cmd[0] = "/bin/ls";
            cmd[1] = path;
            Process p = Runtime.getRuntime().exec(cmd);
            BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
            while (true)
            {
               String line = br.readLine();
               if (line == null)
                  break;
               visitor.visit(directory, line);
            }
            int ev = p.waitFor();
            if (ev != 0)
            {
               throw new IllegalStateException("/bin/ls exit value: " + ev);
            }
         }
         catch (Exception exc)
         {
            throw new IllegalStateException(exc);
         }
      }
   }
}

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #1 - Posted 2009-12-01 16:44:25 »

I woulda thought their listFiles() method used the same underlying code as dir and ls but never underestimate Java for getting these little things wrong for 10 years... Smiley

Cas Smiley

Online Riven
« League of Dukes »

JGO Overlord


Medals: 748
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #2 - Posted 2009-12-01 16:50:50 »

These are the results for a directory with 65536 files:

strategy   first result   last result
File.list()7010ms7010ms
DirectoryIterator.iterate(visitor)   210ms7780ms


Each test was performed after a reboot, as the 2nd invocation of File.list() takes only 110ms, due to OS caching.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Online Riven
« League of Dukes »

JGO Overlord


Medals: 748
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #3 - Posted 2009-12-01 17:44:21 »

Both versions seem to have problems with spaces in the path.
I tried the obvious stuff with quotes, but it didn't work out.
Will fix soon, I hope.


Fixed!

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline i30817

Junior Member





« Reply #4 - Posted 2009-12-01 20:24:53 »

Whats this?

Quote
line = line.substring(36); // offset / magic value

I actually have this stupid thing, even with the sort the file access is what kills it,
besides the directory / not directory comparation.

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
    /**
     * Returns a flat view of a file hierarchy.
     *
     * The file list returns the files descending from sum,
     * including any in the sum array. They come
     * sorted by natural ordering for their
     * filenames for each directory only.
     * The directory list returns the directories in the given hierarchy
     * including any in the sum array. The directories are not sorted.
     * @param levels
     * @param sum
     * @param files
     * @param directories
     */

    public static void getFiles(int levels, File[] sum, List<File> files, List<File> directories) {
        Comparator<File> orderFiles = new Comparator<File>() {

            @Override
            public int compare(File o1, File o2) {
                return Strings.compareNatural(o1.getName(), o2.getName());
            }
        };

        Arrays.sort(sum, orderFiles);

        getFiles(levels, sum, files, directories, orderFiles);
    }

    private static void getFiles(int levels, File[] sum, List<File> files, List<File> directories, Comparator<File> comp) {
        List<File[]> subFilesList = new ArrayList<File[]>(50);
        //O(n^2.log n)
       for (File f : sum) {
            File[] subFiles = f.listFiles();
            if (subFiles == null) {
                files.add(f);
            } else {
                directories.add(f);
                Arrays.sort(subFiles, comp);
                subFilesList.add(subFiles);
            }
        }

        if (levels > 0) {
            for (File[] subFiles : subFilesList) {
                getFiles(levels - 1, subFiles, files, directories, comp);
            }
        }
    }
Online Riven
« League of Dukes »

JGO Overlord


Medals: 748
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #5 - Posted 2009-12-01 21:17:08 »

Whats this?
Quote
line = line.substring(36); // offset / magic value
Just open the command-prompt, type 'dir' and hit enter. You'll see where that 36 comes from.




If you want to recursively iterate (lazily / non-blocking) over a directory-structure:

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
69  
70  
71  
public class FileUtil
{
   public static Iterator<File> getFileHierachyIterator(File dir)
   {
      return FileUtil.getFileHierachyIterator(dir, null);
   }

   public static Iterator<File> getFileHierachyIterator(File dir, FileFilter filter)
   {
      return FileUtil.getFileHierachyIterator(dir, filter, null);
   }

   public static Iterator<File> getFileHierachyIterator(final File dir, final FileFilter filter, final Comparator<File> sorter)
   {
      List<File> files = new ArrayList<File>();

      File[] ff = dir.listFiles(filter);
      if (ff == null) // access denied, very unlikely
        return new ArrayList<File>().iterator();
      for (File file : ff)
         files.add(file);

      if (sorter != null)
         Collections.sort(files, sorter);

      final Iterator<File> it = files.iterator();

      return new Iterator<File>()
      {
         private Iterator<File> subs;

         public boolean hasNext()
         {
            if (subs != null && subs.hasNext())
               return true;
            return it.hasNext();
         }

         public File next()
         {
            if (subs == null || !subs.hasNext())
            {
               File file = it.next();

               if (file.isDirectory())
               {
                  subs = FileUtil.getFileHierachyIterator(file, filter, sorter);
               }
               else
               {
                  subs = null;
               }
               return file;
            }

            if (subs != null)
            {
               if (subs.hasNext())
                  return subs.next();
               subs = null;
            }
            return it.next();
         }

         public void remove()
         {
            throw new UnsupportedOperationException("hm... would you really want that??");
         }
      };
   }
}

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline irreversible_kev

Junior Member





« Reply #6 - Posted 2009-12-01 21:53:22 »

Just open the command-prompt, type 'dir' and hit enter. You'll see where that 36 comes from.

Minor issue but

1  
dir /B

should help you get rid of some code
Online Riven
« League of Dukes »

JGO Overlord


Medals: 748
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #7 - Posted 2009-12-01 21:55:25 »

Minor issue but

1  
dir /B

should help you get rid of some code

Shocked Thanks.


Fixed.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline i30817

Junior Member





« Reply #8 - Posted 2009-12-02 00:17:20 »

BTW, do you have any idea why sorting many files based in lastModified() takes forever and ever? I  already have the files in a map/list like structure, but when actually getting the file time it goes forever and ever.

Any trick for that?
Online Riven
« League of Dukes »

JGO Overlord


Medals: 748
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #9 - Posted 2009-12-02 01:07:37 »

BTW, do you have any idea why sorting many files based in lastModified() takes forever and ever? I  already have the files in a map/list like structure, but when actually getting the file time it goes forever and ever.

Any trick for that?

Because you MUST NOT read the lastModified() again and again for each comparison. Cache it somewhere, as Map<File, Long> will solve it

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline i30817

Junior Member





« Reply #10 - Posted 2009-12-02 01:38:57 »

So obvious. Thanks. No way to avoid that first hit though right?
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

CogWheelz (15 views)
2014-07-30 21:08:39

Riven (21 views)
2014-07-29 18:09:19

Riven (14 views)
2014-07-29 18:08:52

Dwinin (12 views)
2014-07-29 10:59:34

E.R. Fleming (32 views)
2014-07-29 03:07:13

E.R. Fleming (12 views)
2014-07-29 03:06:25

pw (42 views)
2014-07-24 01:59:36

Riven (42 views)
2014-07-23 21:16:32

Riven (30 views)
2014-07-23 21:07:15

Riven (31 views)
2014-07-23 20:56:16
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!