Java-Gaming.org Hi !
Featured games (91)
games approved by the League of Dukes
Games in Showcase (808)
Games in Android Showcase (239)
games submitted by our members
Games in WIP (872)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Files on your harddrive can be mapped and passed directly into textures/buffers  (Read 15883 times)
0 Members and 1 Guest are viewing this topic.
Offline theagentd
« Posted 2015-02-03 17:38:16 »

EDIT:
Don't do this! It has some severe problems which can cause your program to grind into a halt after mapping a lot of files. See the below discussion for more details! I recommend using FileChannels with a temporary direct buffer, not to map them!

Hello.

Today I tried out something interesting. I was working on making it possible to source my streamable textures from multiple sources (asset files, texture packs and raw files in a directory) when I stumbled upon an interesting function. FileChannels allow you to get a MappedByteBuffer for a certain range of a file on your harddrive. This byte buffer is direct, meaning that it's possible to pass it directly into OpenGL functions like glTexImage2D() and glBufferData(). For texture data, this can be pretty worthless. Most of the time the data is stored in some kind of image format (PNG or even JPG) which needs to be decompressed before it can be passed into glTexImage2D(), but for my streaming system this decompression was way too slow. My streamable texture files contain the raw image data compressed using S3TC or BPTC which I simply dump into glCompressedTexImage2D(). To test out the potential gains of using mapped files, I've developed a small test program which compares the CPU performance of 3 different ways of loading raw texture data from a file.

The first way of loading stuff is with old-school input streams. This requires a lot of copies, since FileInputStreams work with byte[]s, not ByteBuffers. We have to read the texture data into a byte[], copy it to a direct ByteBuffer and then pass it to glTexImage2D().
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
   private static byte[] bytes = new byte[DATA_LENGTH];
   private static ByteBuffer buffer = BufferUtils.createByteBuffer(DATA_LENGTH);

   private static long loadStream() throws Exception {
      long startTime = System.nanoTime();

      FileInputStream fis = new FileInputStream(RAW_FILE);
     
      int read = 0;
      while(read < DATA_LENGTH){
         int r = fis.read(bytes, read, DATA_LENGTH-read);
         if(r == -1){
            throw new IOException();
         }
         read += r;
      }
      buffer.put(bytes).flip();
      glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 512, 512, 0, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
     
      fis.close();
     
      return System.nanoTime() - startTime;
   }


The second way is to use NIO and FileChannels. FileChannels work on ByteBuffers directly, so we can use a direct ByteBuffer from the start!
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
   private static ByteBuffer buffer = BufferUtils.createByteBuffer(DATA_LENGTH);

   private static long loadChannel() throws Exception{
      long startTime = System.nanoTime();

      FileInputStream fis = new FileInputStream(RAW_FILE);
      FileChannel fc = fis.getChannel();
     
      while(buffer.hasRemaining()){
         fc.read(buffer);
      }
      buffer.flip();
      glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 512, 512, 0, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
     
      fc.close();
      fis.close();
     
      return System.nanoTime() - startTime;
   }


The last most awesome way is to use NIO and FileChannels to map part of the file as a MappedByteBuffer. This is just so magical and simple.
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
   private static long loadMapped() throws Exception {
      long startTime = System.nanoTime();

      FileInputStream fis = new FileInputStream(RAW_FILE);
      FileChannel fc = fis.getChannel();
     
      MappedByteBuffer mbb = fc.map(MapMode.READ_ONLY, 0, fc.size());
      glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, 512, 512, 0, GL_RGBA, GL_UNSIGNED_BYTE, mbb);
     
      fc.close();
      fis.close();

      return System.nanoTime() - startTime;
   }


As you can see, there's some timing code in each function. The average time taken to do these operations over a few thousand runs in a loop (all reading the same file over and over again, so it's not representative for IO performance, only CPU performance) is listed below:

Stream:  0.657057 ms
Channel: 0.207856 ms
Mapped:  0.169004 ms

So it's not only simple as hell to do, it's faster as well and doesn't require any temporary memory!

Myomyomyo.
Offline lcass
« Reply #1 - Posted 2015-02-03 17:44:56 »

I guess it depends on when you are loading in your textures , for a faster loading time with a huge number of sprites on a game i can see why it would be necessary however on a small project unless you are literally streaming sprites into the game continuously it would be negligable.

Still very cool.
Offline BurntPizza

« JGO Bitwise Duke »


Medals: 486
Exp: 7 years



« Reply #2 - Posted 2015-02-03 17:50:03 »

Probably not applicable to texture streaming (unless you're evil), but this is also a neat way to get pretty fast IPC, as multiple processes can map the same file and thus communicate in that chunk of memory.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline theagentd
« Reply #3 - Posted 2015-02-03 18:12:10 »

Probably not applicable to texture streaming (unless you're evil)...
Why not? My streamable textures are just each mipmap's compressed texture data dumped to a big file. Being able to map a whole mipmap and pass it in sounds immensely useful, and decompressing the texture is simply too slow, especially since S3TC and BPTC textures don't compress very well.

Myomyomyo.
Offline BurntPizza

« JGO Bitwise Duke »


Medals: 486
Exp: 7 years



« Reply #4 - Posted 2015-02-03 18:17:44 »

But when are you passing textures between processes? Threads maybe, but not processes.

Although I have seen some talk about spinning up multiple jvms in this fashion just so that each jit can focus on a core area of the code, etc. and be profitable if the IPC has low-enough overhead (mmap'ed files). Probably only useful in very particular circumstances. Maybe texture decompression from a conventional format would be in this category.
Offline theagentd
« Reply #5 - Posted 2015-02-03 19:00:12 »

But when are you passing textures between processes? Threads maybe, but not processes.

Although I have seen some talk about spinning up multiple jvms in this fashion just so that each jit can focus on a core area of the code, etc. and be profitable if the IPC has low-enough overhead (mmap'ed files). Probably only useful in very particular circumstances. Maybe texture decompression from a conventional format would be in this category.
I'm not following. I just want to read raw data from the harddrive and pass it to OpenGL in the most efficient way possible. Since mapping the file seems like not only the fastest but also the simplest and most memory efficient way, why is it not fit for streaming? I don't have multiple processes.

Myomyomyo.
Offline BurntPizza

« JGO Bitwise Duke »


Medals: 486
Exp: 7 years



« Reply #6 - Posted 2015-02-03 19:03:15 »

I was just pointing out that this is also a really good way to do IPC, for your purpose it's also perfectly good. Sorry. [/massivederailment]
Offline theagentd
« Reply #7 - Posted 2015-02-03 19:22:06 »

Ah, right. =3 No problem.

Myomyomyo.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #8 - Posted 2015-02-03 19:44:29 »

The only issue with MappedByteBuffer is that you cannot unmap them manually: closing the channel is not enough. You have to rely on the GC to collect it, and by that time you may have exhausted valuable memory.

You can use hackery to unmap a buffer, but as always, it's risky shit and may break with any JRE release.
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
   public static void unmap(MappedByteBuffer buffer)
   {
      try
      {
         Method cleanerMethod = buffer.getClass().getMethod("cleaner", new Class[0]);
         cleanerMethod.setAccessible(true);
         Object cleaner = cleanerMethod.invoke(buffer);

         Method cleanMethod = Class.forName("sun.misc.Cleaner").getMethod("clean", new Class[0]);
         cleanMethod.setAccessible(true);
         cleanMethod.invoke(cleaner);
      }
      catch (Throwable t)
      {
         throw new UnsupportedOperationException("arg", t);
      }
   }


You're seriously better off writing a few JNI methods.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline theagentd
« Reply #9 - Posted 2015-02-03 19:48:22 »

Is there a serious risk of a out-of-memory crash unless I do that?

Myomyomyo.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #10 - Posted 2015-02-03 19:49:31 »

Is there a serious risk of a out-of-memory crash unless I do that?
MappedByteBuffer is rarely used due to this design flaw. There really are no valid use-cases for this class - it's like LinkedList: those who use it are not yet properly informed persecutioncomplex

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #11 - Posted 2015-02-03 20:03:37 »

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
public class MappedByteBufferStressTest
{
   public static void main(String[] args) throws IOException
   {
     boolean doUnmap = true;

      for(int i = 0; i < 64; i++)
      {
         File file = new File("D:/stress." + i + ".map");
         System.out.println("Writing 1GB to " + file.getName());
         RandomAccessFile raf = new RandomAccessFile(file, "rw");
         raf.setLength(1024 * 1024 * 1024);

         ByteBuffer tmp = ByteBuffer.allocateDirect(64 * 1024);
         while (tmp.hasRemaining())
            tmp.put((byte) 'a');
         tmp.flip();

         FileChannel fc = raf.getChannel();
         MappedByteBuffer mapped = fc.map(MapMode.READ_WRITE, 0, fc.size());
         while (mapped.hasRemaining())
            mapped.put((ByteBuffer) tmp.clear());
         fc.close();
         raf.close();

         if(doUnmap)
         {
            unmap(mapped); // N.B.: async - the OS will be doing disk I/O long after the unmap, potentially long after your process terminates.
         }
      }
   }

   public static void unmap(MappedByteBuffer buffer)
   {
      try
      {
         Method cleanerMethod = buffer.getClass().getMethod("cleaner", new Class[0]);
         cleanerMethod.setAccessible(true);
         Object cleaner = cleanerMethod.invoke(buffer);

         Method cleanMethod = Class.forName("sun.misc.Cleaner").getMethod("clean", new Class[0]);
         cleanMethod.setAccessible(true);
         cleanMethod.invoke(cleaner);
      }
      catch (Throwable t)
      {
         throw new UnsupportedOperationException("arg", t);
      }
   }
}


With
doUnmap
set to
true
, the OS stresses, yet plows through.

With
doUnmap
set to
false
, the OS freezes/locks up, processes stop responding, until you manage to kill the java process - which might fail, so save your work prior to running this snippet.

To answer your question: it won't be an OutOfMemoryError, but... much worse.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #12 - Posted 2015-02-03 20:25:41 »

Actually I've got a particularly good use for it: I use it as my world-database. It's opened for the duration of the game's execution and makes for a particularly efficient way to manage the data for 4 million territories. Server does the same.

WRT unmapping... if you're wise you're sticking to a specific JVM anyway for deployment: all you need to know is that your hack works for that JVM. One day it may break - but when that day comes you only have to look in this one place and switch back to the filechannel method.

Cas Smiley

Offline theagentd
« Reply #13 - Posted 2015-02-03 20:29:39 »

I'll look into this more and update the first post to reflect this discussion once I've gotten rid of the 25cm of snow that fell outside our house...

Myomyomyo.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #14 - Posted 2015-02-03 20:30:58 »

@Cas: keep in mind that ranges of a MappedBuffer can be moved in and out of memory at any time, which means getting or putting a single byte may cause a >100ms hickup, so I wouldn't access my MappedByteBuffer in a game loop.

* Riven stating the obvious persecutioncomplex

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline princec

« JGO Spiffy Duke »


Medals: 1146
Projects: 3
Exp: 20 years


Eh? Who? What? ... Me?


« Reply #15 - Posted 2015-02-03 20:51:37 »

It's accessed like a database - no realtime access required generally - though when rendering the world screen it does read it every time the view moves. But I can live with hiccups.

Cas Smiley

Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #16 - Posted 2015-02-03 20:54:11 »

On a side note: I saw a lengthy YouTube video in which John Carmack said he ditched mapped files entirely, recently, due to their unpredictable latency. IIRC they were extensively used in Rage.

Sorry for the derail, I'll try to be better next time persecutioncomplex

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #17 - Posted 2015-02-03 21:12:51 »

Some background information: (see: EVALUATION)
http://bugs.java.com/view_bug.do?bug_id=4724038

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline theagentd
« Reply #18 - Posted 2015-02-04 00:55:52 »

Riven, you mentioned writing my own file mapping implementation using JNI. That sounds a bit annoying. Isn't there any simple implementation of it out there that I can use?

Myomyomyo.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #19 - Posted 2015-02-04 09:46:12 »

As Cas said, if you control the environment, the reflection hack is 'good enough'. Obviously you'd cache the Method references to squeeze out the 'last' bit of performance - Method.invoke itself is probably fast enough for your (considerable) requirements.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline basil_

« JGO Bitwise Duke »


Medals: 418
Exp: 13 years



« Reply #20 - Posted 2015-02-05 23:15:39 »

a simpler way to clean things is
1  
( (DirectBuffer)buffer ).cleaner().clean();
yet a bad idea since it may not be implemented in some VMs.

*edit*

i also stopped using mapped buffers. ran into trouble with big files and out of memory issues. using streams in separated threads works out much better.
Pages: [1]
  ignore  |  Print  
 
 

 
Riven (846 views)
2019-09-04 15:33:17

hadezbladez (5790 views)
2018-11-16 13:46:03

hadezbladez (2602 views)
2018-11-16 13:41:33

hadezbladez (6206 views)
2018-11-16 13:35:35

hadezbladez (1498 views)
2018-11-16 13:32:03

EgonOlsen (4733 views)
2018-06-10 19:43:48

EgonOlsen (5791 views)
2018-06-10 19:43:44

EgonOlsen (3275 views)
2018-06-10 19:43:20

DesertCoockie (4174 views)
2018-05-13 18:23:11

nelsongames (5500 views)
2018-04-24 18:15:36
A NON-ideal modular configuration for Eclipse with JavaFX
by philfrei
2019-12-19 19:35:12

Java Gaming Resources
by philfrei
2019-05-14 16:15:13

Deployment and Packaging
by philfrei
2019-05-08 15:15:36

Deployment and Packaging
by philfrei
2019-05-08 15:13:34

Deployment and Packaging
by philfrei
2019-02-17 20:25:53

Deployment and Packaging
by mudlee
2018-08-22 18:09:50

Java Gaming Resources
by gouessej
2018-08-22 08:19:41

Deployment and Packaging
by gouessej
2018-08-22 08:04:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!