Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (477)
Games in Android Showcase (107)
games submitted by our members
Games in WIP (536)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: 1 [2]
  ignore  |  Print  
  Loading a binary file  (Read 5938 times)
0 Members and 1 Guest are viewing this topic.
Offline overnhet

Junior Member




Java games rock!


« Reply #30 - Posted 2004-03-16 11:47:08 »

Yep, direct access to struct field is purely a matter of taste. I think that (some) structs are "lightweight" classes used as mere data storage mean and won't benefit much from clean encapsulation.

Now if get/set (or swpalmer's way) are an option, it should be feasible. Did anyone ever used a bytecode generation lib ?
Offline swpalmer

JGO Coder




Where's the Kaboom?


« Reply #31 - Posted 2004-03-16 12:13:13 »

PrinceC it is only an awful travesty because you actually made the mistake of looking at the code Smiley.  If this is auto-generated code from metadata info etc.  Then don't look at it.  Pay no attention to the man behind the curtain.  The public API is decent and that's all you need to care about - so long as the performance is acceptable.

As a first go I would even ditch the offset and start with this:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
public class Vector3f implements Vector3fReader, Vector3fWriter { 
private final ByteBuffer buf;
public Vector3f(ByteBuffer buf) {
  this.buf = buf;
}
public float x() { return buf.getFloat(0); }
public float y() { return buf.getFloat(4); }
public float z() { return buf.getFloat(8); }
public void x(float x) { buf.setFloat(0); }
public void y(float y) { buf.setFloat(4); }
public void z(float z) { buf.setFloat(8); }


But, you want to be able to read a huge number of Vector3f from a single chunk of memory.. that's why you added the offset, right?  But why worry about constructing new Vector3fs to deal with this?  They are small and likely short lived in this context.
I would profile first.. then added the offset idea if it was required.

Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #32 - Posted 2004-03-16 12:57:05 »

If the man behind the curtain is constantly doing 3x as much work as the C++ man then sadly certain operations are going to be a lot slower than in C++, and this particular kind of operation is of especial significance to people performing intensive geometry processing operations.

Creating tons of little objects is sadly still a killer in a rendering loop, and the memory is better used for something else.

Consider my age-old BSP conundrum. You have a BSP file, containing data for vertices, triangles, nodes, etc. If you tried to represent this in Java as an actual graph of Vector3fs and so on you'd end up with 50mb of object header bloat before you even got round to storing the data. This is another problem that the sliding struct solves really neatly.

Cas Smiley

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #33 - Posted 2004-03-16 13:17:35 »

I think it would help get the attention of the different advantages of struct's were better categorized, and it was made clear that they are related to completely separate abstract use-cases.

e.g., Structs:
1. Provide OO access to "raw" data from an external source (usually either  a network-protocol or a file-format)
2. Provide higher-speed access to raw data that is large data structures with many small fixed-size data structures inside them

Sliding Structs:
3. Significantly reduce memory requirements and increase speed for apps that have a huge number of very small objects that cannot effectively be represented as arrays
4. Enable portions of the OO universe to be constrained to a sequential portion of native memory so that the *application* can manually dump and restore them as needed.

NB: I've never looked at using mem-mapped BB's for use-case 4; the last time I was doing that was pre-1.4.x (c.f. below).

I've no idea if these are good categorizations/use-cases, but the current descriptions tend to be different depending upon who you ask, with lots of "Oh, and BTW there's also another good reason", instead of a clear, easy-to-read overview.

Quote

Consider my age-old BSP conundrum. You have a BSP file, containing data for vertices, triangles, nodes, etc. If you tried to represent this in Java as an actual graph of Vector3fs and so on you'd end up with 50mb of object header bloat before you even got round to storing the data.


I've faced the same problem when dealing with massive parse-trees / AST's (of the order of 10^6 - 10^7 (or more) nodes), in the days before BB's existed, and structs would have been a great help (in the end I just borrowed RAM and did partial-evaluations instead). In this example, you also want to "checkpoint" frequently (losing the partial results of calculations that have generated many millions of nodes is not something you want to do!) - and BB-contained sliding-structs (IIRC your definition of the "sliding" struct...) provide a very convenient way of doing this: (temporarily) I don't care about the fileformat, just let me do a straight-through dump-to-disk at maximum speed Grin. If the system crashes, I at least know I can get the data back...

Quote

This is another problem that the sliding struct solves really neatly.


Indeed: it is "another problem".

I'm not saying it's not a worthy problem to solve, but in the current state of things I think the structs issue comes across in a very confused manner to people who don't already know all the advantages.

Describing it as separate issues may also make it easier for sun to evaluate in the light of other activities - e.g. if they are separately spending considerable effort elsewhere trying to make objects have smaller memory footprint then part of the use-cases may be already improved from that different direction.

malloc will be first against the wall when the revolution comes...
Offline Mark Thornton

Senior Member





« Reply #34 - Posted 2004-03-16 13:31:42 »

Quote

Sliding Structs:
3. Significantly reduce memory requirements and increase speed for apps that have a huge number of very small objects that cannot effectively be represented as arrays

The archetypal case for this is probably arrays of Complex values. In my opinion this case should not be dealt with via structs, but rather by one of the immutable object proposals.
The structs for external communication and the need for efficient classes for things like Complex are two separate issues that do not deserve a common solution.
Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #35 - Posted 2004-03-16 14:51:46 »

Quite. Structs aren't meant to be used as lightweight objects, as they must be backed by a byte buffer.

Cas Smiley

Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #36 - Posted 2004-03-16 14:59:52 »

Quote
Quite. Structs aren't meant to be used as lightweight objects, as they must be backed by a byte buffer.

Cas Smiley


1. Then your BSP example is pointless.
2. As I mentioned, in the AST use-case there is a definite advantage in being backed by a BB

malloc will be first against the wall when the revolution comes...
Offline Mark Thornton

Senior Member





« Reply #37 - Posted 2004-03-16 16:05:17 »

Quote
I don't care about the fileformat, just let me do a straight-through dump-to-disk at maximum speed Grin. If the system crashes, I at least know I can get the data back...

Even before the advent of nio, I found dumping data to a file in a binary format was often limited by the disk speed. Depending on the data, piping the stream through a gzip compression would sometimes improve matters further.

Although in the past I have used 'structs' memory mapped in C++, in most cases I now think this is a mistake. For all but the most trivial objects (and objects which must be guaranteed to remain trivial), the lost capability relative to 'proper' objects eventually becomes a problem.
Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #38 - Posted 2004-03-16 16:06:23 »

Why is the BSP example pointless? I'd simply map my BSP file directly from disk into memory and then when I needed to walk it I'd just have a sliding Node struct to follow the tree, and sliding Vector3fs to find coordinates in it, etc.

Cas Smiley

Offline rreyelts

Junior Member




There is nothing Nu under the sun


« Reply #39 - Posted 2004-03-16 16:16:31 »

Quote

1  
2  
3  
4  
5  
6  
7  
8  
public class Vector3f implements Vector3fReader, Vector3fWriter {
private int offset;
private final ByteBuffer buf;
private static final int X_OFFSET = 0;
private static final int Y_OFFSET = 4;
private static final int Z_OFFSET = 8;
private static final int SIZE = 12;
// ...


Holy crap. That code is so close in style and structure to my own that somebody would swear that you copied it from me or vice-versa. (A good argument against these ridiculous software patents). The largest difference is that I've got some more complicated indexing to do in some places, like:

1  
2  
3  
4  
5  
6  
7  
8  
9  
public Link linkAt( Link link, int index ) {
  long nodeLinkPtr = getPtr( network.nodes, offset + LINK_OFFSET );
  int nodeLinkStartIndex = ( int ) ( nodeLinkPtr - network.nodeLinksAddress );
  assert( nodeLinkStartIndex >= 0 );
  int nodeLinkIndex = nodeLinkStartIndex + index * SIZEOF_INT;
  int linkIndex = network.nodeLinks.getInt( nodeLinkIndex );
  assert( linkIndex >= 0 );
  return link.setIndex( linkIndex );
}

God bless,
-Toby Reyelts

About me: http://jroller.com/page/rreyelts
Jace - Easier JNI: http://jace.reyelts.com/jace
Retroweaver - Compile on JDK1.5, and deploy on 1.4: http://retroweaver.sf.net.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline rreyelts

Junior Member




There is nothing Nu under the sun


« Reply #40 - Posted 2004-03-16 16:46:59 »

Quote
Yes, that's exactly the point of structs - directly modifying data in Buffers by mapping normal Java fields over them.


I was looking closer at your proposal Cas, and one of the problems I have is that the Struct class can't have non-buffer data members. Wouldn't we better off adding some kind of field access specifier or field metadata attribute - like @structmember, that indicates that a field is actually read from the buffer? The structmember attribute could be a little more flexible too - like specifying the number of bytes that get read from the buffer for that data member.

God bless,
-Toby Reyelts

About me: http://jroller.com/page/rreyelts
Jace - Easier JNI: http://jace.reyelts.com/jace
Retroweaver - Compile on JDK1.5, and deploy on 1.4: http://retroweaver.sf.net.
Offline rreyelts

Junior Member




There is nothing Nu under the sun


« Reply #41 - Posted 2004-03-16 17:15:58 »

Quote
I'm waiting for someone to come up with an implementation.. /me taps fingers impatiently...


Hey swpalmer,

You could probably do something like:

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
MyStruct extends Struct {

  private @structfield( 1, 4 ) long val1;
  private @structfield( 2 ) int val2;
  private @structfield( 3 ) int val3;

  // some other fields

  public long val1() {
    return val1;
  }

  public val1( long val1 ) {
    this.val1= val1;
  }
}


Some thoughts:

1) You need the structfield attribute to tell you what order the fields should be read/written, because fields aren't necessarily stored in order in bytecode (unfortunately).

2) Like Cas said, this won't remove any of the array-bounds checks that exist in the current code. It'll just make things a little simpler.

This would probably be pretty simple code to write. Maybe a couple days worth of effort.

God bless,
-Toby Reyelts

About me: http://jroller.com/page/rreyelts
Jace - Easier JNI: http://jace.reyelts.com/jace
Retroweaver - Compile on JDK1.5, and deploy on 1.4: http://retroweaver.sf.net.
Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #42 - Posted 2004-03-16 17:24:37 »

I reckon that the whole struct thing can be done with metadata and a specially tuned VM now. I don't quite know exactly how metadata works yet but I have a hunch it does what we want, ie. mark a class as having its fields laid out in memory in order, etc. and the VM can generate special case code when it encounters the metadata tags.

Is there anywhere that explains 1.5 metadata and how to use it on the web that's easy to get at?

Cas Smiley

Offline Jeff

JGO Coder




Got any cats?


« Reply #43 - Posted 2004-03-16 17:35:31 »

Quote
, but I would still expect the JVM to deal with it and produce correct results by generating code that manipulated non-native-endian mapped fields according to the JLS instead of just getting it wrong.


Im not sure I understand you.  From what I THINK I understand, this IS what Java does today.  if you use DataInput/DataOutput then it twiddles the bytes to be a standard that any Java VM can correctly read in.

Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Offline rreyelts

Junior Member




There is nothing Nu under the sun


« Reply #44 - Posted 2004-03-16 17:54:16 »

Quote
I reckon that the whole struct thing can be done with metadata and a specially tuned VM now. I don't quite know exactly how metadata works yet but I have a hunch it does what we want, ie. mark a class as having its fields laid out in memory in order, etc. and the VM can generate special case code when it encounters the metadata tags.

Is there anywhere that explains 1.5 metadata and how to use it on the web that's easy to get at?


Sun is very reluctant to attach any semantics to metadata outside of emitting compiler warnings or errors when it comes to the Java compiler and VM:

http://forum.java.sun.com/thread.jsp?forum=316&thread=431398&message=1957008#1957008

If you want information about metadata, you should download the 2.4 prototype, which contains draft specs for all sorts of things, including JSR 175:

http://java.sun.com/developer/earlyAccess/adding_generics/

God bless,
-Toby Reyelts

About me: http://jroller.com/page/rreyelts
Jace - Easier JNI: http://jace.reyelts.com/jace
Retroweaver - Compile on JDK1.5, and deploy on 1.4: http://retroweaver.sf.net.
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #45 - Posted 2004-03-16 18:42:39 »

Quote
Why is the BSP example pointless? I'd simply map my BSP file directly from disk into memory and then when I needed to walk it I'd just have a sliding Node struct to follow the tree, and sliding Vector3fs to find coordinates in it, etc.

Cas Smiley


You said structs were not to be used for lw objects. I assumed you would want the whole BSP in memory at once, in which case I'd expect you to be using lw objects.

But it hadn't occurred to me that you might want to only selectively/lazily load the file.

How tricky is it to implement deformable maps in this scenario? (and I don't mean just a single elevator, or on/off hole-in-the-wall. I mean interesting changes Smiley)

malloc will be first against the wall when the revolution comes...
Offline swpalmer

JGO Coder




Where's the Kaboom?


« Reply #46 - Posted 2004-03-16 20:59:03 »

Quote
You said structs were not to be used for lw objects. I assumed you would want the whole BSP in memory at once, in which case I'd expect you to be using lw objects.

But it hadn't occurred to me that you might want to only selectively/lazily load the file.

I think Cas wants to reuse a single 'struct' to give some "structure" to different areas of the ByteBuffer on-the-fly.  This isn't about many objects or lazy loading.  It's just about making the data accessible in an efficient way. It is simply the "view" of the data in the ByteBuffer that we want control over.
(I'm sure Cas will correct me if I got that wrong.)

Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #47 - Posted 2004-03-17 08:19:02 »

Yes, that's what I, and many games developers, need to be able to do. There are so many overheads to doing it with Java objects it's just not feasible.

There are plenty of other uses too such as a vertex buffer processor. Say you needed to write interleaved data out to an OpenGL vertex buffer. You might have tons and tons of different data to deal with and you can't very well store it all in objects and keep reading and writing objects and constructing and waiting for GC as it's discarded every frame for a completely fresh set. Far better just to point a struct at it and slide it along the buffer to do your processing.

Cas Smiley

Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #48 - Posted 2004-03-17 08:30:56 »

So how is that different from the efficiencies gained from immutable objects?

malloc will be first against the wall when the revolution comes...
Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #49 - Posted 2004-03-17 09:10:37 »

Because we're talking about data that's so mutable you process megabytes of it 85 times a second. Or data that's so large and complex that you'd need to double the memory storage to use it in Java and pointlessly get the garbage collector to traverse the whole thing periodically only to discover it's all still referenced. Or data that takes ages and ages to load in using the Serializable interface because it's so big but is mapped in the blink of an eye with a MappedByteBuffer.

Cas Smiley

Offline vrm

Junior Member




where I should sign ?


« Reply #50 - Posted 2004-03-17 09:23:03 »

there is type problem too, how annoying is to manipulate C unsigned char with java signed bytes ...
Offline Mark Thornton

Senior Member





« Reply #51 - Posted 2004-03-17 10:13:07 »

Quote
Or data that takes ages and ages to load in using the Serializable interface because it's so big but is mapped in the blink of an eye with a MappedByteBuffer.

The mapping may not take long, but if you then go and read that data sequentially from beginning to end, it can be slower than using ordinary file read operations. This happens when the mapped pages remain in memory after you have been past them and cause parts of your heap to be kicked out to swap file. So memory mapping is great when you randomly read only a small part of the data, but if you actually read the whole lot it can be quite bad (at least on Windows in my experience).
In my case I have  600MB heap and a 1GB data file. That brings me quite close to the address space available to ordinary applications on 32 bit windows.
You also have noticed my RFE relating to some of the problems with memory mapping:
http://developer.java.sun.com/developer/bugParade/bugs/4724038.html
Pages: 1 [2]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Riven (20 views)
2014-07-29 18:09:19

Riven (13 views)
2014-07-29 18:08:52

Dwinin (12 views)
2014-07-29 10:59:34

E.R. Fleming (31 views)
2014-07-29 03:07:13

E.R. Fleming (12 views)
2014-07-29 03:06:25

pw (42 views)
2014-07-24 01:59:36

Riven (42 views)
2014-07-23 21:16:32

Riven (28 views)
2014-07-23 21:07:15

Riven (29 views)
2014-07-23 20:56:16

ctomni231 (60 views)
2014-07-18 06:55:21
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!