Java-Gaming.org Hi !
Featured games (83)
games approved by the League of Dukes
Games in Showcase (538)
Games in Android Showcase (132)
games submitted by our members
Games in WIP (600)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  On disk NIO buffer  (Read 2257 times)
0 Members and 1 Guest are viewing this topic.
Offline gouessej
« Posted 2012-08-30 22:15:36 »

Hi

I have to use an API to export some meshes in a particular format. It uses NIO buffers in input. The problem is that the memory footprint of JFPSM is already very high. I would like to use a NIO buffer whose content is on disk and that loads some parts of it into memory only when needed so that it lowers my memory footprint if I avoid using a relative bulk get() call on the whole buffer. I think I can't do that with MappedByteBuffer. Is there already a class doing that? I think that EhCache is a bit too much for my needs, isn't it?

Best regards.

Offline Riven
« League of Dukes »

« JGO Overlord »


Medals: 840
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #1 - Posted 2012-08-30 22:33:37 »

MappedByteBuffer seems perfectly capable of doing that (until you go over ~2GB)

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social
Offline gouessej
« Reply #2 - Posted 2012-08-30 23:22:16 »

JFPSM cannot yet be used on low end 32 bits cheap laptops. I don't want to set Xmx or XX:MaxDirectMemorySize to more than 256 MB which is impossible with my main level. MappedByteBuffer uses a direct NIO buffer and there is no way of controlling the size of the data really resident in physical memory. I would like to have only at most 4 KB resident data in physical memory at a time.

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Riven
« League of Dukes »

« JGO Overlord »


Medals: 840
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #3 - Posted 2012-08-30 23:35:57 »

MappedByteBuffers let the OS manage the memory for you. If the OS is stressed for memory, it will simply release the memory in the MappedByteBuffer. That's all done behind the scenes, mainly because you don't need to know, and the OS is smart enough to do the mosts efficient thing, most of the time.



TL;DR:
Use MappedByteBuffers and stop worrying Pointing

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social
Offline princec

« JGO Spiffy Duke »


Medals: 429
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #4 - Posted 2012-08-30 23:37:35 »

Alternatively why not just use a random access file? Why the strange need for NIO buffering?

Cas Smiley

Offline gouessej
« Reply #5 - Posted 2012-08-31 19:11:45 »

Alternatively why not just use a random access file? Why the strange need for NIO buffering?
The API uses NIO buffers, I can only pass NIO buffers...

I will try to reduce the memory footprint of my whole application before trying to use on disk buffers.

Offline Riven
« League of Dukes »

« JGO Overlord »


Medals: 840
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #6 - Posted 2012-08-31 19:13:20 »

Nothing prevents you from pushing a byte[] into a ByteBuffer.


I will try to reduce the memory footprint of my whole application before trying to use on disk buffers.
As said, the OS manages the memory for you, there is no need to worry about it.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social
Offline gouessej
« Reply #7 - Posted 2012-08-31 19:22:37 »

Nothing prevents you from pushing a byte[] into a ByteBuffer.


I will try to reduce the memory footprint of my whole application before trying to use on disk buffers.
As said, the OS manages the memory for you, there is no need to worry about it.
The data sets will be too big to fit into a single buffer. The file in input occupies about 250 MB but the meshes in output might occupy tens of GB (Paris metropolitan region in 3D). MappedByteBuffer uses a direct NIO buffer under the hood, it means that I will have to destroy it by myself if the Java heap is not stressed enough, it's a known problem but I'm even not sure such a buffer could be allocated in some cases. I have found several things to optimize in Ardor3D:
http://www.ardor3d.com/forums/viewtopic.php?f=10&t=5905#p16504

I have to map only a part of the data into physical memory at a time and to avoid trying to get a backing array or the whole content of the buffer.

Offline gouessej
« Reply #8 - Posted 2012-08-31 20:29:03 »

The file in input is here. You can try to convert it into a more appropriate format for rendering by using OSMToWorld.

Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

rwatson462 (29 views)
2014-12-15 09:26:44

Mr.CodeIt (20 views)
2014-12-14 19:50:38

BurntPizza (40 views)
2014-12-09 22:41:13

BurntPizza (75 views)
2014-12-08 04:46:31

JscottyBieshaar (37 views)
2014-12-05 12:39:02

SHC (50 views)
2014-12-03 16:27:13

CopyableCougar4 (45 views)
2014-11-29 21:32:03

toopeicgaming1999 (113 views)
2014-11-26 15:22:04

toopeicgaming1999 (100 views)
2014-11-26 15:20:36

toopeicgaming1999 (30 views)
2014-11-26 15:20:08
Understanding relations between setOrigin, setScale and setPosition in libGdx
by mbabuskov
2014-10-09 22:35:00

Definite guide to supporting multiple device resolutions on Android (2014)
by mbabuskov
2014-10-02 22:36:02

List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!