Java-Gaming.org    
Featured games (91)
games approved by the League of Dukes
Games in Showcase (577)
games submitted by our members
Games in WIP (498)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1] 2
  ignore  |  Print  
  Loading a binary file  (Read 5538 times)
0 Members and 1 Guest are viewing this topic.
Offline nonnus29

Senior Member




Giving Java a second chance after ludumdare fiasco


« Posted 2004-02-07 03:24:04 »

One of the big problems with Java is loading a binary file; for example an md2 model or a tga file.  I 've been reading alot of code from the internet lately and it seems the common way to do it is load the file as a byte buffer then cast the bits to the data type you need and fill in our values as you go.  This sucks compared to the super easy and simple c method of simply casting a chunk of memory to a struct.  So is there a better way to do it?  Maybe with nio?
Offline K.I.L.E.R

Senior Member




Java games rock!


« Reply #1 - Posted 2004-02-07 07:21:47 »

I myself have been wandering the same thing for ages.
Unfortunately the method you mentioned seems to be the only method I've seen.

Alternatives? Don't see any.

Vorax:
Is there a name for a "redneck" programmer?

Jeff:
Unemployed. Wink
Offline kevglass

JGO Kernel


Medals: 85
Projects: 25


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #2 - Posted 2004-02-07 07:33:41 »

Depends what you're looking for, speed wise there isn't anything as nice a mapping structs over a block of memory.

If you're an easy way to manipulate a block of data I've found it easier in the past to wrap up the byte array in an input stream and wrap that in a data input stream.

Just writing a read method on each of your objects normally covers this. So you just end up with TGAHeader object that knows how to read itself from a data block.

Kev

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #3 - Posted 2004-02-07 08:21:00 »

Java Land is a warm and fluffy place, where cuddly Objects live in happiness and the Sun never sets (ho ho). Just beyond the borders of Happy Java Land there lies the nasty, spiky, rocky, thorny place called External Systems Land, and it is here that you find all the other programs written in the world by anyone else, ever. Unfortunately Java Land is a tiny little country surrounded on all sides by External Systems Land.

But the benevolent ruler of Java Land has decided that because Java Land is so great, no-one need ever leave, or talk to anyone else outside of the border in their own tongue, and when the Objects do try and leave, they are only allowed to take a hammer with them and are under strict instructions to use said hammer to solve any problems they come across. Rather like America but with hammers instead of M16s.

What this colourful analogy means to you is - because the binary world of external systems is totally incompatible with the managed object heirarchies in Java, the APIs available to you are extremely inefficient at talking to external systems, and torturously complicated if you want to try and make them efficient. Wouldn't you love to be able to simply map a bytebuffer using NIO onto a BSP file, and simply access it using a BSP class, and it'd Just Work like in C++?

Well - you could do this, but it requires a new feature in the compiler and the VM. Head on down to Bug Parade and Vote for Structs! This feature I consider so important that when I discovered generics would be available in 1.5 I immediately shifted all 3 of my generics votes onto 3 votes for Structs.

Structs are a great solution to this problem as they deal with all the issues of the day neatly:

1. At the end of the day, you're still dealing with Objects, so no existing APIs or GC policy needs changing whatsoever

2. Super-fast and efficient access can be optimised into the JVM, but the compiler can be made to generate current-standards compatible code as a fallback

3. And finally it's the first time Java has officially tried to communicate with external systems in their own tongue

Vote! Vote! Vote!

Cas Smiley

Offline K.I.L.E.R

Senior Member




Java games rock!


« Reply #4 - Posted 2004-02-08 09:43:33 »

In C++ structs have been replaced with enums.

Java 1.5 has enums, why do you want a struct when something better is in place?

Vorax:
Is there a name for a "redneck" programmer?

Jeff:
Unemployed. Wink
Offline Jacko

Junior Member





« Reply #5 - Posted 2004-02-08 09:59:32 »

how does an enum help with reading a binary file?
Offline abies

Senior Member





« Reply #6 - Posted 2004-02-08 10:04:30 »

Quote
In C++ structs have been replaced with enums.
Java 1.5 has enums, why do you want a struct when something better is in place?


In Polish, we have an idiom (which is of course almost meaningless after translation to english): "What a cinderbread has in common with windmill ?" And no, nobody reply 'flour', it is an idiom.

How can you replace struct with enum Huh This is like replacing integers with loops or HashMap with RemoteException. Totally different uses.

Artur Biesiadowski
Offline Orangy Tang

JGO Kernel


Medals: 51
Projects: 11


Monkey for a head


« Reply #7 - Posted 2004-02-08 10:13:37 »

Quote
In C++ structs have been replaced with enums.
Java 1.5 has enums, why do you want a struct when something better is in place?

Eh? The closest thing to a C struct in C++ is surely a class, compilers even treat them identically ('cept for default public/private irregularity). The snag in Java being that you have no control over how things are laid out in memory, so you can't just bung raw binary data straight into an object.

[ TriangularPixels.com - Play Growth Spurt, Rescue Squad and Snowman Village ] [ Rebirth - game resource library ]
Offline cfmdobbie

Senior Member




Who, me?


« Reply #8 - Posted 2004-02-08 13:41:23 »

This is nothing to do with platform-independence!  You continue writing your portable applications, they will not change.

It is possible to implement structs in a completely platform-independent way - some consideration for data type sizes and use of the NIO ByteOrder system will be needed.  The only concern I have for structs is how to handle unsigned data, but that's what discussion is for!

And to address your concerns that it's just not needed - while you can import and manipulate binary data without the structs concept, you might just find that things will be a hell of a lot faster if you could use it.  We don't need Collections, generics, static import, NIO, autoboxing, Javadoc, enumerations, more than one integral and one floting-point type, Swing, JoGL etc.  We don't need them, but they do make the platform a lot nicer/easier/faster.

N.B. My three votes are in for structs, but I don't think it's going to change anything.  Sun are flat-out ignoring this RFE, and I suspect they are hoping it won't gather much interest and will eventually disappear from view.  We're coming up to ONE YEAR of it being in the top 25 list, and still no Sun evaluation.  Gosh.

Hellomynameis Charlie Dobbie.
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #9 - Posted 2004-02-08 19:30:39 »

If you ever really, truly believe in platform independence you will realise that all other platforms that exist aren't Java, and if you want to interoperate - and you sure better do, because there's no point in writing programs that live in their own world these days! - you need the tools to do it in your development platform of choice.

Structs aren't about games programming. They're about efficient I/O with external systems in those external systems' formats. If you can't do that in Java then a powerful reason for using Java, that it is crossplatform and easy to develop in, is moot, as it's far easier just to keep on using the old native code to manipulate the native data.

Put it this way: with structs you would be able to very efficiently write a TCP/IP stack in pure Java, or a networked filesystem, or a harddisk controller, or an OpenGL *driver*, or a 3D game engine. It enables the entire world of external systems programming to be within reach of Java.

Cas Smiley

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline swpalmer

JGO Coder




Where's the Kaboom?


« Reply #10 - Posted 2004-02-08 23:19:31 »

Well, aside from the concept of structs which is all very wonderful.  The way you do this stuff most efficiently NOW does seem to be with ByteBuffers.

You can read in the number of bytes that make up your 'struct' into a ByteBuffer.  You must explicitly set the ByteOrder of the ByteBuffer to match that of your data file.  Then you use get() getInt(), getDouble(), getShort()... to initialize class members.
It is tedious, but I bet you could use reflection to automate most of it... then it would be less tedious - just slow Smiley

Instead of structs it might even make sense to develop a simple API that works now using some sort of mapping to members and a method that initializes an object using that mapping and a ByteBuffer.

Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #11 - Posted 2004-02-09 07:50:54 »

Structs live only inside ByteBuffers. That was the reason they existed!

The whole mapping-things-to-bytebuffers idea is basically fraught with hassles and very inefficient in a JVM, requiring many, many boundschecks, and also many, many lines of code to achieve something which could be very simply achieved with some language enhancements.

Structs is all about making things easier for developers that need to do this kind of stuff. There are a surprising number of developers who need this functionality; the problem is that the vast majority of Java developers that exist right now are ones who have come from a completely different application domain and so when you say "we need structs in the language!" they say "Huh? Why? What's the point in that?"

Cas Smiley

Offline overnhet

Junior Member




Java games rock!


« Reply #12 - Posted 2004-02-09 10:52:00 »

I got that problem too. I tried to use reflection to automate reads from a DataInputStream, as swpalmer suggests in his post, but this failed because :

* Class.getDeclaredFields is not bound to return the fields in any particular order, so there is no reliable way to know what field should be read first ;

* unsigned types cause troubles : if you store an ushort in an int, reflection will see an int and read 4 bytes instead of 2 ;

I also had problems with array lengthes, but I can't remember whether I solved them or not Smiley

Now I am wondering if annotations could be used to solve those issues. Since it is a new feature, I don't fully understand how to use them, but runtime accessible annotations could probably contain information on the order and sign of the fields.

As of performance issues, one could analyze the structure of a class only once, and generate optimized bytecode to read / write the class in binary. Never made that before, but there are a few good tutorials on bytecode modification libraries.
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #13 - Posted 2004-02-09 11:03:15 »

Quote
If you ever really, truly believe in platform independence you will realise that all other platforms that exist aren't Java, and if you want to interoperate
...
Structs [are] about efficient I/O with external systems in those external systems' formats.


...I wish you'd described them like this way back when you first brought them up Smiley (a year ago?).

"Structs as a means of communicating with external binary systems in a neat way" is an argument I can agree with Smiley. Your early descriptions had me thinking of you wanting them mainly for internal JVM use (...I still don't see very compelling reasons for them purely for internal use...).

Although this quote misses your other key point, perhaps it should say: "...external systems' formats, in a fully OOP manner without yuckiness, and with all the maintenance advantages of OOP" Smiley.

malloc will be first against the wall when the revolution comes...
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #14 - Posted 2004-02-09 14:05:59 »

That's basically what the RFE describes. Damn, I thought you'd understood what I was after from the beginning Smiley Go place your 3 votes now!

Cas Smiley

Offline cfmdobbie

Senior Member




Who, me?


« Reply #15 - Posted 2004-02-10 12:38:41 »

Handy link for those who otherwise wouldn't vote because they don't know where to look:

http://developer.java.sun.com/developer/bugParade/bugs/4820062.html

Grin

Hellomynameis Charlie Dobbie.
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #16 - Posted 2004-02-10 14:01:17 »

Thanks Charlie Smiley

Cas Smiley

Offline jherber

Junior Member




Java games rock!


« Reply #17 - Posted 2004-03-14 00:05:05 »

jean-marie dautelle has a very handy set of classes to do exactly what you are asking:  http://jade.dautelle.com/

jim
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #18 - Posted 2004-03-14 11:08:51 »

Almost but not quite. There's no performance gain: multiple bounds checks still, multiple gets and sets with all the associated performance loss Sad

It strikes me that the new Metadata feature in 1.5 might be a good way to implement Structs. More investigation required.

Cas Smiley

Offline Jeff

JGO Coder




Got any cats?


« Reply #19 - Posted 2004-03-15 21:46:16 »

Cas is half right.  Javaland is a happy palce where people play together.  the outside world is an evil places where NOTHING is compatable.

Now in english:

Java defines important standards across systems such as endianess and the size of types.

The rest of the world doesnt have this.  If you took an arbitrary structure and wrote it out from C under Win32 and read it back into a C program under Sparc you would find the structure hopelessly garbled.

This is the fundamental problem.  Although the Java side of the equation isd defined the "binary"  (or C) side of the equation mist emphatically is not.  Therefor any attempt to automatically read such into Java is doomed to failure.  We provide tools for doing thinsg like setting endianess conversions for ByteBuffers but you need to provide the info on how all  those conversions need to be done because there IS no apriori answer.

And thats why it looks the way it does.


Got a question about Java and game programming?  Just new to the Java Game Development Community?  Try my FAQ.  Its likely you'll learn something!

http://wiki.java.net/bin/view/Games/JeffFAQ
Offline rreyelts

Junior Member




There is nothing Nu under the sun


« Reply #20 - Posted 2004-03-15 23:32:22 »

Quote
This sucks compared to the super easy and simple c method of simply casting a chunk of memory to a struct.

If you're doing that to read/write a file, you're playing with fire. Totally ignoring the fact that your code won't run on architectures with different endianness, the in-memory layout of a struct is highly dependent on the compiler you used and the options you use with that compiler.

God bless,
-Toby Reyelts

About me: http://jroller.com/page/rreyelts
Jace - Easier JNI: http://jace.reyelts.com/jace
Retroweaver - Compile on JDK1.5, and deploy on 1.4: http://retroweaver.sf.net.
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #21 - Posted 2004-03-16 07:30:02 »

This isn't the problem that most of us are trying to solve in Java right now. Most of us are trying to interface directly with a known memory layout... if that memory layout is a non-native endianness then no amount of cleverness is going to make it particularly easy for the JVM engineers without resorting to lots of JVM hacking and performance loss anyway, but I would still expect the JVM to deal with it and produce correct results by generating code that manipulated non-native-endian mapped fields according to the JLS instead of just getting it wrong.

At the end of the day any solution presented must be representable in the first instance in pure Java bytecode and execute correctly. The JVM should be able to use its special knowledge of the semantics to eliminate memory copies and unnecessary bounds checks. Then it'd do everything C can do, except reliably Cheesy

Cas Smiley

Offline Mark Thornton

Senior Member





« Reply #22 - Posted 2004-03-16 07:53:11 »

Quote
Almost but not quite. There's no performance gain: multiple bounds checks still, multiple gets and sets with all the associated performance loss Sad

A JVM could recognise a structure like the Dautelle 'struct' class and eliminate most of the bounds checks. I think there is a much better chance of a JRE for this succeeding than for a language change.
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #23 - Posted 2004-03-16 09:13:51 »

The resistance to language change in Java is over and over producing verbose, nasty code and hacks. How long did it take to get "enums" despite the fact that "you can do them in Java anyway just by typing loads and loads of tedious error-prone code"? Enums are in the same league as structs.

Cas Smiley

Offline overnhet

Junior Member




Java games rock!


« Reply #24 - Posted 2004-03-16 10:03:17 »

AFAIK C# has structs and using unsafe code you can read them directly from a memory block. That kind of features should help game developers moving from C++ to .NET (and I doubt they will even consider Java as an alternative). Angry

To Cas : I have spent the last few days playing with metadata, reflection and generics to get my own "poor man's struct" impl. There are still a lot of things to do, and I had to restrict the problem to dumb I/O with Streams and Buffers (no direct mapping to ByteBuffer so far).

There is one thing I wanted to ask about your struct proposal. Suppose I have a simple Vertex3f struct, with x,y,z fields being public : if an instance v is mapped on a ByteBuffer, should v.x = aFloat; modify the underlying ByteBuffer ?
Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #25 - Posted 2004-03-16 10:14:39 »

Yes, that's exactly the point of structs - directly modifying data in Buffers by mapping normal Java fields over them.

Cas Smiley

Offline overnhet

Junior Member




Java games rock!


« Reply #26 - Posted 2004-03-16 10:26:33 »

Then I think it is not feasible with current Java, even with metadata : AFAIK there is no method that would allow to "catch" direct access to a public field and translate it into a ByteBuffer.get() call.

An alternative would be to enforce the use of getter/setter to access struct fields, but who wants to write a bloated v.getX() when a plain v.x is much more readable ?
Offline blahblahblahh

JGO Coder


Medals: 1


http://t-machine.org


« Reply #27 - Posted 2004-03-16 11:03:48 »

Quote

An alternative would be to enforce the use of getter/setter to access struct fields, but who wants to write a bloated v.getX() when a plain v.x is much more readable ?


Unless I'm missing something about performance here (and, let me be clear: as far as performance is concerned I personally couldn't care less for structs: I just want clean OO access to raw data), let's not worry too much about personal preferences for the avoidance of getters/setters...not being able to use direct variable access is not a major barrier to using structs (nb: assuming intelligent compiler I would personally almost always use methods for structs in preference to var access!)

Although (as I said in the getters/setters thread) I'm getting fed up with the extra typing Wink, it is with good reason that the method-based approach is "preferred" (in well-designed and well-implemented projects) for non-local variable access.

malloc will be first against the wall when the revolution comes...
Offline swpalmer

JGO Coder




Where's the Kaboom?


« Reply #28 - Posted 2004-03-16 11:22:33 »

Save some some typing, drop the get/set prefix, like much of the ByteBuffer methods do (e.g. position).
Overloading gets you what you want.

v.x() to get
v.x(Float) to set

Then hopefully the 1.5 metadata stuff can be used to automatically generate the boilerplate code that will redirect these calls to the underlying ByteBuffer.

I'm waiting for someone to come up with an implementation.. /me taps fingers impatiently...

Offline princec

JGO Kernel


Medals: 282
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #29 - Posted 2004-03-16 11:32:23 »

By all means make the fields private and put getters and setters all over them if you need to. The issue is with internal operations on the data, not external access to data, eg. simple vector class:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
public struct Vector3f implements Vector3fReader, Vector3fWriter{
private float x, y, z;
public float length() {
return (float) Math.sqrt(x*x + y*y + z*z);
}
public float getX() { return x; }
public float getY() { return y; }
public float getZ() { return z; }
public void setX(float x) { this.x = x; }
public void setY(float y) { this.y = y; }
public void setZ(float z) { this.z = z; }
}

as opposed to this awful travesty:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
public class Vector3f implements Vector3fReader, Vector3fWriter {
private int offset;
private final ByteBuffer buf;
private static final int X_OFFSET = 0;
private static final int Y_OFFSET = 4;
private static final int Z_OFFSET = 8;
private static final int SIZE = 12;
public Vector3f(ByteBuffer buf) {
this.buf = buf;
}
public void setOffset(int offset) {
if (offset < 0 || offset >= buf.remaining() - SIZE) {
throw new IndexOutOfBoundsException();
}
this.offset = offset;
}
public float length() {
float x = getX(), y = getY(), z = getZ();
return (float) Math.sqrt(x*x + y*y + z*z);
}
public float getX() { return buf.getFloat(offset + X_OFFSET); }
public float getY() { return buf.getFloat(offset + Y_OFFSET); }
public float getZ() { return buf.getFloat(offset + Z_OFFSET); }
public void setX(float x) { buf.setFloat(offset + X_OFFSET, x); }
public void setY(float y) { buf.setFloat(offset + Y_OFFSET, y); }
public void setZ(float z) { buf.setFloat(offset + Z_OFFSET, z); }
}



Cas Smiley

Pages: [1] 2
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

xsi3rr4x (25 views)
2014-04-15 18:08:23

BurntPizza (20 views)
2014-04-15 03:46:01

UprightPath (35 views)
2014-04-14 17:39:50

UprightPath (18 views)
2014-04-14 17:35:47

Porlus (34 views)
2014-04-14 15:48:38

tom_mai78101 (60 views)
2014-04-10 04:04:31

BurntPizza (118 views)
2014-04-08 23:06:04

tom_mai78101 (218 views)
2014-04-05 13:34:39

trollwarrior1 (185 views)
2014-04-04 12:06:45

CJLetsGame (192 views)
2014-04-01 02:16:10
List of Learning Resources
by Longarmx
2014-04-08 03:14:44

Good Examples
by matheus23
2014-04-05 13:51:37

Good Examples
by Grunnt
2014-04-03 15:48:46

Good Examples
by Grunnt
2014-04-03 15:48:37

Good Examples
by matheus23
2014-04-01 18:40:51

Good Examples
by matheus23
2014-04-01 18:40:34

Anonymous/Local/Inner class gotchas
by Roquen
2014-03-11 15:22:30

Anonymous/Local/Inner class gotchas
by Roquen
2014-03-11 15:05:20
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!