Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (477)
Games in Android Showcase (107)
games submitted by our members
Games in WIP (534)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Creating a chunk-based custom binary file format  (Read 798 times)
0 Members and 1 Guest are viewing this topic.
Offline tom_mai78101
« Posted 2014-05-27 11:03:17 »

I wanted to try learning how to create a custom chunk-based binary file. I wrote a small tree structure that implements chunk hierarchy in my documentation:



As I started to work on this, I noticed that I can't come up with a good way to write chunks into byte arrays, then output the byte arrays using DataOutputStream.

I do know that chunk-based binary files contains chunk headers that can be used as indexes in a book. You skip parts of the file until you reach the chunk you wanted, and read in the data from there.

Can the experts teach me how to write chunk-based binary files using DataOutputStream? Thanks in advance.

Offline Abuse

JGO Coder


Medals: 11


falling into the abyss of reality


« Reply #1 - Posted 2014-05-27 12:09:27 »

As I started to work on this, I noticed that I can't come up with a good way to write chunks into byte arrays, then output the byte arrays using DataOutputStream.

You're going to have to be more specific; what is the problem?

All I can suggest is that you look at the file format specification for a popular chunk based format, I'd suggest PNG (section 4.7 onwards); it's straight forward and well documented.

Make Elite IV:Dangerous happen! Pledge your backing at KICKSTARTER here! https://dl.dropbox.com/u/54785909/EliteIVsmaller.png
Offline tom_mai78101
« Reply #2 - Posted 2014-05-27 13:32:54 »

As I started to work on this, I noticed that I can't come up with a good way to write chunks into byte arrays, then output the byte arrays using DataOutputStream.

You're going to have to be more specific; what is the problem?

I created 2 classes, Chunk (superclass) and Header (subclass). Both of these are used to test out the creation of a custom chunk-based binary file. My plan is to just write the Header into the created file, and then load from the created file. This is to test and see if reading chunks and writing chunks are working or not.

Chunk:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
69  
70  
71  
72  
73  
74  
75  
76  
77  
package saving;

import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;

public class Chunk {
   public static final long HEADER_SIGNATURE = 0x4845414445520000L; //ASCII code for HEADER00 (2 zeros at end)
  public static final int SIGNATURE_COUNT = 1;
   
   protected long signature;    //8 bytes of readable tag label.
  protected int size;         //Size of chunk.
  protected long[] chunkSignatureList; //Array of signatures that the file has saved.
 
   public Chunk() {
      this.signature = 0L;
      this.size = 0;
      this.chunkSignatureList = new long[SIGNATURE_COUNT]; //This number is the total number of signatures.
  }
   
   public Chunk(Chunk chunk) {
      this.signature = chunk.getSignature();
      this.size = chunk.getSize();
   }
   
   public void read(DataInputStream input) {
      try {
         this.signature = input.readLong();
         this.size = input.readInt();
      }
      catch (IOException e) {
         throw new RuntimeException("Error in chunk reading the input", e);
      }
   }
   
   public void write(DataOutputStream output) {
      try {
         output.writeLong(signature);
         output.writeInt(size);
      }
      catch (IOException e) {
         throw new RuntimeException("Error in chunk writing the input", e);
      }
     
   }
   
   public long[] getChunkSignatureList() {
      return chunkSignatureList;
   }
   
   public long getSignature() {
      return signature;
   }
   
   public void setSignature(long sig) {
      this.signature = sig;
   }
   
   public int getSize() {
      return size;
   }
   
   public void setSize(int size) {
      this.size = size;
   }
   
   public void setChunkSignatureList(long[] list) {
      this.chunkSignatureList = list;
   }
   
   public static Chunk convert(Chunk chunk) {
      long sig = chunk.getSignature();
      if (sig == HEADER_SIGNATURE)
         return new Header((Header) chunk);
      return null;
   }
}


Header:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
package saving;

import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;

public class Header extends Chunk {
   private long fileTypeName;      //8 bytes containing the name of the program that reads this file.
  private int fileFormat;         //1 byte containing a period. 3 bytes containing the format name.
 
   public Header() {
      super();
      this.signature = HEADER_SIGNATURE;
      this.fileTypeName = 0L;
      this.fileFormat = 0;
   }
   
   public Header(Header header) {
      super(header);
      this.signature = HEADER_SIGNATURE;
      this.fileTypeName = header.getFileTypeName();
      this.fileFormat = header.getFileFormat();
      this.chunkSignatureList = header.getChunkSignatureList();
   }
   
   @Override
   public void read(DataInputStream input) {
      super.read(input);
      try {
         this.fileTypeName = input.readLong();
         this.fileFormat = input.readInt();
      }
      catch (IOException e) {
         throw new RuntimeException("Can't read header.", e);
      }
   }
   
   @Override
   public void write(DataOutputStream output) {
      super.write(output);
      try {
         output.writeLong(this.fileTypeName);
         output.writeInt(this.fileFormat);
      }
      catch (IOException e) {
         throw new RuntimeException("Can't write header.", e);
      }
   }
   
   public long getFileTypeName() {
      return this.fileTypeName;
   }
   
   public int getFileFormat() {
      return this.fileFormat;
   }
   
   public void setFileTypeName(long name) {
      this.fileTypeName = name;
   }
   
   public void setFileFormat(int format) {
      this.fileFormat = format;
   }
}


  • When you start writing to a file, where do you create the chunks first, before writing it?
  • What needs to be done before reading it back in?
  • I have heard of using the interface
    Serialization
    , is this good to use?
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline cylab

JGO Ninja


Medals: 38



« Reply #3 - Posted 2014-05-27 14:31:53 »

I wouldnt go through the hassle of creating a custom binary format. Just use xstream and save xml or json directly from your data model: http://xstream.codehaus.org/tutorial.html

Mathias - I Know What [you] Did Last Summer!
Offline tom_mai78101
« Reply #4 - Posted 2014-05-27 14:44:45 »

Maybe each chunk contains the tag of the current chunk, the tag of the next chunk, and the size of the current chunk. The DataInputStream should read the tags of the current chunk, check to see if it matches the signature that is required. If not, go read in the next tag, then skip the current chunk via the given size of the current chunk, and then repeat by matching the required signature with the next tag that was last read in.

The only problem is that I don't know how to tell the DataInputStream to go jump back and forth between the chunks, each time checking the tags until the signature was found, and read in that chunk. Unlike in C, C++, where given an integer pointer, one can just tell the pointer to go back X amount of bytes in a byte array (or set the pointer to the first element of a byte array) and repeat the steps to check on the tag.

I wouldnt go through the hassle of creating a custom binary format. Just use xstream and save xml or json directly from your data model: http://xstream.codehaus.org/tutorial.html

Even though it really is a hassle, it is something that I need to conquer in the near future. More like skipping out from using libraries.

Offline Roquen
« Reply #5 - Posted 2014-05-27 15:12:57 »

Quote
I wouldnt go through the hassle of creating a custom binary format.
My experience is that it's more hassle to create a so-called human readable format using a library (or not) than a simple custom binary format.

To OP: I'd say there are two types of file formats:  Interchange and custom.  The former is for allowing multiple people to write programs than can handle data it knows (or simply cares) about and ignore the rest.  The latter is only used by one program and any associated tools.  Why are you writing an interchange format?
Offline tom_mai78101
« Reply #6 - Posted 2014-05-27 15:41:08 »

To OP: I'd say there are two types of file formats:  Interchange and custom.  The former is for allowing multiple people to write programs than can handle data it knows (or simply cares) about and ignore the rest.  The latter is only used by one program and any associated tools.  Why are you writing an interchange format?

It's the first thing that came to my mind when I started on planning out the file format. I didn't know much about it, other than that it looks nice for a documentation.

I actually tried to figure out how to load that sort of "interchange" file format with the following code. I haven't tested it out, or do anything with it other than following along with my "logic".

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
   public static void load(Game game, String filename) {
      File load = new File(filename);
      if (load.isFile()) {
         int chunkLoadedCount = 0;
         byte[] buffer = null;
         ArrayList<byte[]> buffers = new ArrayList<byte[]>();
         DataInputStream input = null;
         try {
            //TODO: Modify this number so that it can load even more chunks.
           while (chunkLoadedCount < 1) {
               try {
                  input = new DataInputStream(new BufferedInputStream(new FileInputStream(load)));
                 
                  while (true) {
                     //This loop allows the possibility to read each "chunk" and take actions depending on the "chunk's" signature.
                    //If it fails, it redo the file loading until the total count of loaded chunks have reached its goal.
                    try {
                        int sig = input.readInt();
                        int size = input.readInt();
                       
                        //TODO: Create a new signature tag. There won't be any "chunks" structures.
                       if (sig != 0xA1A1A1A1) {
                           input.skip(size);
                        }
                        else {
                           buffer = new byte[size];
                           input.read(buffer);
                           buffers.add(buffer);
                           chunkLoadedCount++;
                        }
                     }
                     catch (Exception e) {
                        throw new Exception(e);
                     }
                  }
               }
               catch (Exception e) {
                  break;
               }
            }
         }
         catch (FileNotFoundException e) {
            throw new RuntimeException("Something with the file not being found.", e);
         }
         catch (IOException e) {
            throw new RuntimeException("Something with the file not being read correctly.", e);
         }
         finally {
            try {
               input.close();
            }
            catch (IOException e) {
               throw new RuntimeException("Unable to close the save file correctly.", e);
            }
         }
         handleLoadedBuffers(game, buffers);
      }
   }


1. To Roquen, what can you suggest me to do? I love hearing some recommendations on this.
2. To everyone, would the code above work out for the file format I had planned before Roquen's post?
Offline princec

JGO Kernel


Medals: 342
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #7 - Posted 2014-05-27 16:29:39 »

Screw the file format!

Replace the lot with an embedded HSQLDB engine using cached tables. One database per "world". That way the bulk of your file format is managed by HSQLDB, and also the indexing and access.

Cas Smiley

Offline CHmSID

Senior Newbie


Exp: 7-9 months



« Reply #8 - Posted 2014-05-28 00:46:53 »

Is there any specific reason why you want to use DataInputStream over RandomAccessFile? In RandomAccessFile you have a method seek() which sets the pointer in the files to anywhere you want.
Offline tom_mai78101
« Reply #9 - Posted 2014-05-28 01:42:43 »

Screw the file format!

Replace the lot with an embedded HSQLDB engine using cached tables. One database per "world". That way the bulk of your file format is managed by HSQLDB, and also the indexing and access.

Cas Smiley

Do I really have to implement a SQL-variant database manager into my game? I felt like that is overkill.

Is there any specific reason why you want to use DataInputStream over RandomAccessFile? In RandomAccessFile you have a method seek() which sets the pointer in the files to anywhere you want.

Really? Oooh~~! I didn't know that.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Roquen
« Reply #10 - Posted 2014-05-28 09:30:39 »

I'm a DB idiot so the comment of Cas might as well be written in <fill in the name of some dialect that hasn't been spoke in a thousand years>.

One potential advantage of interchange style formats can be evolution of the file format.  So not version N vs. N+1, but you're at N and working on what will be in N+1.  I don't find so, but they do have some merit here.

One possible style simply adapting what you already have would be something like this (in-post, probably doesn't even compile).

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
69  
70  
71  
72  
73  
74  
75  
76  
  public class SaveGame {
    DataInputStream src;
    int version;
    Game game;
   
    // for sanity checking and debugging
   void checkTag(int expected, String error) throws Exception
    {
      int tag = src.readInt();
     
      if (tag == expected)
        return;
     
      throw new Exception("whatever" + error); // whatever exception type
   }
   
    public static final int HEADER = 0x00000000; // whatever
   
    public static void read(Game game, String filename) throws Exception
    {
      File load = new File(filename);
      SaveGame sg = new SaveGame();
     
      if (load.isFile()) {
        sg.src     = new DataInputStream(new BufferedInputStream(new FileInputStream(load)));
        sg.checkTag(HEADER, "whatever");
        sg.version = sg.src.readInt();
        sg.game    = game;
        Game.read(sg);
      }
    }
   
    // mock classes
   
    public static class Game {
      public static void read(SaveGame sg) throws Exception
      {
        Player.read(sg);
        Entity.read(sg);
        // whatever else in version 1
       
        if (sg.version > 1) {
         // read in all extra stuff in version 2, etc
       }
      }
    }
   
    public static class Player {
      private static final int TAG = 0x00000000; // whatever
     
      public static void read(SaveGame sg) throws Exception
      {
        sg.checkTag(TAG, "player");
        // read in all stuff in version 1
       
        if (sg.version > 1) {
          // read in all extra stuff in version 2, etc
       }
      }
    }
   
   
    public static class Entity {
      private static final int TAG = 0x00000000; // whatever
     
      public static void read(SaveGame sg) throws Exception
      {
        sg.checkTag(TAG, "entity");
        // read in all stuff in version 1
       
        if (sg.version > 1) {
          // read in all extra stuff in version 2, etc
       }
      }
    }
  }
Offline cylab

JGO Ninja


Medals: 38



« Reply #11 - Posted 2014-05-28 10:14:22 »

Quote
I wouldnt go through the hassle of creating a custom binary format.
My experience is that it's more hassle to create a so-called human readable format using a library (or not) than a simple custom binary format.
I dont want to push this since the OP explicitely wants a custom binary format, but I dont get your point.  Whats the hassle in serializing a selfcontained datamodel to any format using a library like xstream. Sure you need to define your datamodel, the used types, values and references, with care, but how is that more of a hassle than defining a binary format?

Mathias - I Know What [you] Did Last Summer!
Offline princec

JGO Kernel


Medals: 342
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #12 - Posted 2014-05-28 10:40:16 »

Screw the file format!

Replace the lot with an embedded HSQLDB engine using cached tables. One database per "world". That way the bulk of your file format is managed by HSQLDB, and also the indexing and access.
Do I really have to implement a SQL-variant database manager into my game? I felt like that is overkill.
The reasons to do it are:
1. It works
2. It does what you want
3. It takes care of the binary format for you
4. It provides an upgrade path for your metadata (DDL - alter table)
5. It adds a bunch of handy extra stuff like ACID - much more agreeable than trashing your data after an exception in the middle of a write, no?
6. The jar is small
7. It's easy to use

Cas Smiley

Offline Roquen
« Reply #13 - Posted 2014-05-28 10:54:41 »

@cylab: As always stuff like this is use-case dependent.  I very rarely have a one-to-one mapping of memory representation vs. storage.  Simple examples:  Data elements which I want to be lossless might require extra work on my part to be lossless in a "human-readable" format.  Like floating point elements.  Others I'm happy (or even desirable) to have a lossy storage.
Offline tom_mai78101
« Reply #14 - Posted 2014-05-28 11:07:20 »

@Roquen:

So, interchange formats are like formats that evolve over time? Seems applicable for an in-dev project, right?

Screw the file format!

Replace the lot with an embedded HSQLDB engine using cached tables. One database per "world". That way the bulk of your file format is managed by HSQLDB, and also the indexing and access.
Do I really have to implement a SQL-variant database manager into my game? I felt like that is overkill.
The reasons to do it are:
1. It works
2. It does what you want
3. It takes care of the binary format for you
4. It provides an upgrade path for your metadata (DDL - alter table)
5. It adds a bunch of handy extra stuff like ACID - much more agreeable than trashing your data after an exception in the middle of a write, no?
6. The jar is small
7. It's easy to use

Cas Smiley

Hm...

*rubs moustache and chin heavily*

Could consider it for next project. Cheesy
Pages: [1]
  ignore  |  Print  
 
 

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

pw (35 views)
2014-07-24 01:59:36

Riven (33 views)
2014-07-23 21:16:32

Riven (21 views)
2014-07-23 21:07:15

Riven (24 views)
2014-07-23 20:56:16

ctomni231 (55 views)
2014-07-18 06:55:21

Zero Volt (47 views)
2014-07-17 23:47:54

danieldean (38 views)
2014-07-17 23:41:23

MustardPeter (43 views)
2014-07-16 23:30:00

Cero (59 views)
2014-07-16 00:42:17

Riven (56 views)
2014-07-14 18:02:53
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!