Java-Gaming.org Hi !
Featured games (90)
games approved by the League of Dukes
Games in Showcase (741)
Games in Android Showcase (225)
games submitted by our members
Games in WIP (823)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  XM File Formats  (Read 13225 times)
0 Members and 1 Guest are viewing this topic.
Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Posted 2016-07-01 19:31:42 »

I have a thread in General Discussion where I post general updates to the project. I wanted to start a thread here containing project-specific things that don't involve updates. I.E. discussion, but I am sharing the code of course. I want to break the old thread.

As most of you know, I have been making file formats. X-Media is what XM stands for. I have an image file, mipmap file, and a packed file which contains multiple XM files. (well any file that is smaller than Integer.MAX_VALUE) That is a general run down of what I have and it is pretty much complete!

I would really like if someone were to test out my library and benchmark the loading times against your current file loading method. This would be an amazing help to the development. I am also open to suggestions on the project. I am talking practical use. It's not hard to use this library at all.

I have uploaded the whole source code, a XM File Convertor, and also a snapshot of the current jar at My github XM File Format Repository

The library is self explanitory, but I have dedicated a few hours to writing wiki pages... which turned out great! The code is fully commented. Check it out Smiley


You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Offline HeroesGraveDev

JGO Kernel


Medals: 382
Projects: 11
Exp: 4 years


┬─┬ノ(ಠ_ಠノ)(╯°□°)╯︵ ┻━┻


« Reply #1 - Posted 2016-07-01 23:32:37 »

The consecutive post limit was designed to stop people WIP-spamming when nobody else was interested in discussing the project. If you find yourself in this situation then you need a blog rather than a forum thread.

It was not designed so people would instead spam a bunch of new threads once they hit the limit.

There are plenty of free blogging options that will be better suited to your project.

Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Reply #2 - Posted 2016-07-02 01:05:58 »

Limit? It's in release. ._. it is moved to a more appropriate place.

You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline 65K
« Reply #3 - Posted 2016-07-02 12:33:21 »

So, the big question is, why would anyone want to use a new file format ?
From looking at your Wiki, I don't get it. Are you saying its better to download uncompressed files to save decompression time ? Then, you do seem to have compressed file formats as well ?
Adding your own benchmark results would help as starting point.

Lethal Running - a RPG about a deadly game show held in a futuristic dysoptian society.
Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Reply #4 - Posted 2016-07-02 17:36:52 »

Hi 65K

I will try and reword the wiki more direct. I just spewed information.

1  
2  
- The goal of the file formats are to be decompressed at loading time.
+ The goal of the XMI, XMM file formats are to not be decompressed at loading time.

This saves the decompression overhead a PNG file would have to go under. There is optional compression. The only time you compress these files are to transfer them over, say the internet. There is a utility that packs and unpacks the files. The result is an XMP file.

See the Quick Start Guide on readme.md

The reason to use my file format is just that - speed of loading. It is DEFINITELY better than using PNG files. You can see a list of advantages by file format on the wiki called Advantages. Please note I kept XMM file out of the bunch, because it is basically a pack of XMI images to be used as mipmap levels.

The Advantages article touches on DDS files - the biggest competitor. DDS files are stored on the GPU then decompressed and read there. The bandwidth is shorter, but there is still the overhead of decompression. DDS files have lossy compression methods, which are terrible for gradients... so I read. While the GPUs are powerful, there is no need to do compression.

If not use XM files, use Direct Draw Surface files.

In time, XM is going to feature a heightmap file and as of about 10 minutes ago, a moving image - gif style.

You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Offline DarkCart

JGO Kernel


Medals: 121
Projects: 9
Exp: 50 years


It's all in the mind, y'know.


« Reply #5 - Posted 2016-07-02 19:21:04 »


*snip*

The goal of the file formats are to be decompressed at loading time. This saves the decompression overhead a PNG file would have to go under. There is optional compression. The only time you compress these files are to transfer them over, say the internet. There is a utility that packs and unpacks the files. The result is an XMP file.


The reason to use my file format is just that - speed of loading. It is DEFINITELY better than using PNG files. You can see a list of advantages by file format on the wiki called Advantages. Please note I kept XMM file out of the bunch, because it is basically a pack of XMI images to be used as mipmap levels.

*snip*

So let me get this straight. The only thing that your format does better than PNG is that it's slightly faster to load and decompresses at loading time?

This may only be true for me but I've never had to wait for a PNG to load in my game, so speed doesn't matter. I don't get how the second one makes it better.

So at the end of the day you really just cloned PNG.

The darkest of carts.
Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Reply #6 - Posted 2016-07-02 19:31:01 »

No DarkCart,

My file format is always uncompressed. When you transfer over the network the files are compressed to save data. Otherwise, they are not compressed. PNG files are compressed, loaded in then uncompressed then used. XMI files are uncompressed, loaded in and used.

Its not a PNG clone. There are 5 different file formats here, too. XMI is the base for the images.

You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Offline CopyableCougar4
« Reply #7 - Posted 2016-07-02 21:39:30 »

The only way to convince others to use your formats is by showing benchmarks for loading time as well as on-disk size comparison.

Either wandering the forum or programming. Most likely the latter Smiley

Github: http://github.com/CopyableCougar4
Offline Archive
« Reply #8 - Posted 2016-07-03 00:10:38 »

The whole reason why many popular image formats use compression is to save space. When it comes to games, imagine a modern AAA game that takes up about 6GB of disk space with compression. With your formats, you'll end up using probably 8-12GB depending on how many images were in the game. They might as well use a BMP file because those are not compressed.

Offline CoDi^R
« Reply #9 - Posted 2016-07-03 00:30:58 »

tl&dr: You are solving the wrong problem.

Quote from: ... said (almost) no coder ever
Loading PNG is too damn slow! It also has way too many options!

Know what? It doesn't matter. If you don't want to decompress, use plain TGA or BMP. In your Wiki you reason that the BMP format has a brazenly large header of ~50 bytes? You just insulted my 4 GZ PC with a SSD and plenty of RAM, and he demands an apology.

My mobile phone told me that it happily trades a few milliseconds of load time for less SD storage any day.

You wrote of checks about face culling (what?) and translucency. There is some context which dictates whether an image must be rendered with alpha-blending or alpha-masking enabled. This is not required to be part of the image data. If your game needs to scan pixels to determine if some of them are non-opaque, you are doing it wrong.

Also, mipmap levels. Please describe a use case where each mipmap level needs its own "transparency" flag. Moreover, last time I worked with mipmaps, the <size of level N> was simply <size of level N-1>/2. So, the "minimal" data you would actually need to store is <size of the topmost level> plus <number of mipmap levels>.

Endianness: There are two major choices, and you take the one 99% of the consumer hardware out there does not use. That's ok, doesn't matter much if you handle it carefully. What also doesn't (should not) matter is whether image data is stored RGBA, ARGB or BGRA. You just need to be consistent. In case of doubt this is - at the latest - just sorted out by some vec4 swizzling after texture sampling in the pixel shader.

Quote from: ... said no artist ever
Hey cool, a custom image format! No plugin for my painting app yet? No problemo, bro, I'll use that nifty command line converter of yours!

One day, you'll not be working in a vacuum anymore. Artists draw pretty pictures. They want to update and get them into the game pretty fast, and pretty often. They want to do that in a tool they know, with exporters and options they know.

They also want to browse and look at them comfortably, be it with their nice gallery app or just in a file explorer.

They do not want to run arcane bash commands or batch files.

<rant /off>

So where are we now?

As it happens, I can see a reason why one would use such a "family of file formats": as some form of intermediate data, e.g. for caching, or (pre-)transforming image data into something friendly to the target platform. E.g. some asset build which takes an image and converts to DXT5 for PC, and ETC2 or whatever they use nowadays on mobile. Strictly automated, no front-end of any kind, hands-off for any non-programmers.

But even in that case, I'd rather happily trade some milliseconds and use a more flexible solution.

ps: the real slim shady XM

Robotality - steamworks4j - @code_disaster - codi^r @ #java-gaming
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Slyth2727
« Reply #10 - Posted 2016-07-03 01:02:06 »

You guys have to chill. He's working on a cool project that he obviously enjoys. No point in telling him nobody will ever use it, he's enjoying coding something. That should be enough for you guys.
It's like saying "Hey, don't show me or continue that art project! Nobody will ever buy it, geez." Who cares? If you don't care about the project don't post on it.

The advice that doesn't say that is great though. CoDi^r's response was great.
Offline theagentd
« Reply #11 - Posted 2016-07-03 02:19:59 »

You guys have to chill. He's working on a cool project that he obviously enjoys. No point in telling him nobody will ever use it, he's enjoying coding something.
The beef this entire community has with Hydroque is that he's saying the exact opposite of that. He's saying he puts the benefit of the masses as his top priority.

Myomyomyo.
Offline HeroesGraveDev

JGO Kernel


Medals: 382
Projects: 11
Exp: 4 years


┬─┬ノ(ಠ_ಠノ)(╯°□°)╯︵ ┻━┻


« Reply #12 - Posted 2016-07-03 02:41:46 »

You guys have to chill. He's working on a cool project that he obviously enjoys. No point in telling him nobody will ever use it, he's enjoying coding something. That should be enough for you guys.
It's like saying "Hey, don't show me or continue that art project! Nobody will ever buy it, geez." Who cares? If you don't care about the project don't post on it.

The advice that doesn't say that is great though. CoDi^r's response was great.

I don't think that Hydroque should neccessarily discontinue the project. I'm just saying that if you've posted 5 times in a row and nobody's responded, then maybe these forums aren't the right place to discuss the project. And even in the situations where you're not getting responses but people seem interested, there are proper ways to go about bypassing that limit (message a mod, or find someone who's interested and ask them to comment). Creating a new thread to continue the WIP-spamming is somewhat rude.

Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Reply #13 - Posted 2016-07-03 06:43:00 »

Know what? It doesn't matter. If you don't want to decompress, use plain TGA or BMP. In your Wiki you reason that the BMP format has a brazenly large header of ~50 bytes? You just insulted my 4 GZ PC with a SSD and plenty of RAM, and he demands an apology.
You over exaggerated. It has a large header of 54 bytes, 4 bytes should do. Its old. Thats one part of the argument that goes with color table look up, supporting all BMP file formats, and the crappy storage of the pixels thereof. Not to mention BGR. BMP shouldn't be used, and you complaining about your precious HDD space is a little odd in your following text.

My mobile phone told me that it happily trades a few milliseconds of load time for less SD storage any day.
I would gladly choose a PNG file for phone... or DDS. Yeah DDS.

You wrote of checks about face culling (what?) and translucency. There is some context which dictates whether an image must be rendered with alpha-blending or alpha-masking enabled. This is not required to be part of the image data. If your game needs to scan pixels to determine if some of them are non-opaque, you are doing it wrong.
To elaborate, when you want face culling on plants and objects, using face culling is something to think about. When you enable culling for plants, you get a funky 2/4 face plant with culling on. This simple little boolean allows checks for just that. Using OpenGL's alpha enables isn't a good thing to do. Discarding pixels in shaders so it goes.

Also, mipmap levels. Please describe a use case where each mipmap level needs its own "transparency" flag. Moreover, last time I worked with mipmaps, the <size of level N> was simply <size of level N-1>/2.
You are right. Each mipmap level doesn't need a transparency boolean on its own. There is the global one after all. So I guess I will rip that right out of there Smiley I am not sure what you were referring to with the size. You never negate one when multiplying to figure out how many bytes are used. This isn't array offsets.

In case of doubt this is - at the latest - just sorted out by some vec4 swizzling after texture sampling in the pixel shader.
That is if you use all one image file format indeed. That argument doesn't mean much.

One day, you'll not be working in a vacuum anymore. Artists draw pretty pictures. They want to update and get them into the game pretty fast, and pretty often. They want to do that in a tool they know, with exporters and options they know.
True. You do realize I will be writing plugins and such for these formats?

Lol funny. XM is just a title. The real formats aren't even XM. XMI, XMM, XMP, XMH, and XMV.

Quote from: theadgentd
He's saying he puts the benefit of the masses as his top priority.
Actually, I am an entirely selfless person with no more interest than to give my everything to people. ._.
...and I never said that.

Quote from: HeroesGraveDev
I'm just saying that if you've posted 5 times in a row and nobody's responded, then maybe these forums aren't the right place to discuss the project.
Thanks for letting me know that WIP threads have a post limit before you can't post anymore. I didn't know this, and I won't be posted any WIP threads. I made this thread in a more specific place. No conversation sparked over there because I typed a well essay. It wasn't my intent to make another WIP thread.

Nonetheless the project is good for at least me. It is of benefit to me. Therefore I shall use it and expand on it as I go. Its not a bad format, and once I write a few different things for editors all will be cool.

You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Offline CoDi^R
« Reply #14 - Posted 2016-07-03 07:40:41 »

Quote from: Hydroque
You over exaggerated. It has a large header of 54 bytes, 4 bytes should do. Its old. Thats one part of the argument that goes with color table look up, supporting all BMP file formats, and the crappy storage of the pixels thereof. Not to mention BGR. BMP shouldn't be used, and you complaining about your precious HDD space is a little odd in your following text.

Please read again. I do not complain about HDD space at all. I complain about trying to solve a problem I don't agree we even have. CPUs are fast (decoding at load time), SSDs are freaking fast (seeking over a few extra header bytes), and there is plenty of RAM (if some is even needed).

Quote from: Hydroque
To elaborate, when you want face culling on plants and objects, using face culling is something to think about. When you enable culling for plants, you get a funky 2/4 face plant with culling on. This simple little boolean allows checks for just that.

Again, wrong problem space. Face culling is a geometry property, not one of the texture painted on that geometry.

Quote from: Hydroque
I am not sure what you were referring to with the size. You never negate one when multiplying to figure out how many bytes are used. This isn't array offsets.

A mipmap is a series of images. It starts with the original image, level #0. Level #1 contains the same image at half the size (width/2, height/2) of the original. Level #2 contains the same image at half the size (width/2, height/2) of level #1.

Quote from: Hydroque
True. You do realize I will be writing plugins and such for these formats?

Yes, I've realized.

Robotality - steamworks4j - @code_disaster - codi^r @ #java-gaming
Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Reply #15 - Posted 2016-07-03 09:47:12 »

You guys are just mad because I have a good file format that you didn't think of.

You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Offline 65K
« Reply #16 - Posted 2016-07-03 10:00:10 »

The reason to use my file format is just that - speed of loading. It is DEFINITELY better than using PNG files.
Then why don't just prove it with proper benchmarks ?
Without benchmarking such discussions are pointless.

Lethal Running - a RPG about a deadly game show held in a futuristic dysoptian society.
Offline Icecore
« Reply #17 - Posted 2016-07-03 12:23:15 »

I have a good file format that you didn't think of.
Pack Raw Data to zip is you’re format?)
sorry but you not create not Zip algorithm, not PNG - you can't сall them Yours
Framework - yes, Format - no

One story: Team Lead from my old work - Copy Past VP8 codec, rename output extension to “crapextension”
And gloriously talking everybody that hi create mega cool Video packing format.

and Another one: when i looking for job one company talking that they have mega cool own language like php  
like php – means they rename php files to “VAV” extension and fools all clients =)

offtop:
I find them Wink
looks like they finaly rename "vav" to "htm", but look on blog post aligns
http://www.vis-design.com/en/blog/
and this ppl make Web page for president and banks XD

If you have question why them? – because corruption, mafia and many-many lies ^^
Nah.. noting special in many countries same (if not even all Sad)

Last known State: Reassembled in Cyberspace
End Transmission....
..
.
Offline ziozio
« Reply #18 - Posted 2016-07-03 16:20:16 »

Hydroque - No one is jealous of your file format, no one is gutted they didn't think of it before. No one is going to use this format.

People are just surprised of the arrogance you have in thinking you have created something new that the masses will want to use because you haven't. Please don't stop developing your format, you obviously feel very proud for what you have done and you should continue having this enthusiasm when furthering your project.
Offline HeroesGraveDev

JGO Kernel


Medals: 382
Projects: 11
Exp: 4 years


┬─┬ノ(ಠ_ಠノ)(╯°□°)╯︵ ┻━┻


« Reply #19 - Posted 2016-07-03 23:56:28 »

You guys are just mad because I have a good file format that you didn't think of.

Hahahahaha. @Longor1996 beat you by almost 3 years, and he used this unoriginal format for a legitimate purpose, rather than advertising it as better than PNG.

Please just quit the arrogance. You're only making yourself look bad.

Offline Abuse

JGO Ninja


Medals: 60


falling into the abyss of reality


« Reply #20 - Posted 2016-07-04 00:22:14 »

Stop wasting your time reinventing the wheel (badly); write some games instead!

:edit:

...and for the record, PNG images don't have to be compressed.

Back when I was doing J2ME work for MIDP1 devices a neat way of doing image rotation was to:
- store the png images uncompressed inside the jar (relying on the jar's own deflate to keep the file size reasonable)
- load each into a byte[]
- rotate the uncompressed pixels in the IDAT chunk in situ (never supported multiple chunks, so image size was constrained)
- recalculate the CRC
- then pass the byte[] to the midp1 image loader.

A horrible hack, but useful knowledge as it forced me to become intimately familiar with the excellent PNG format.
Offline trollwarrior1
« Reply #21 - Posted 2016-07-04 06:35:31 »

This is getting out of hand.. Seriously. At first I though OP was just doing practice, but it turns out he is serious about his 'file format'.

There is no need to replace PNG with uncompressed file format. The file size difference is huge. Your files will go up on average 4 times in size. The whole reason PNG is used is because it is the fastest lossless compression format with good compression ratio that also supports multiple pixel formats. Not to mention that since it has been a standard for many years now it is supported on many platforms and all the tools export / import PNG.

You haven't invented anything. In fact, you took a step backwards and now you are trying to prove that the correct way to make a wheel is to make it a square. If you are really concerned about PNG decode time, I have a much better proposal for you. Use regular PNG like everybody else. Then, during runtime, after you read your PNG, save it's uncompressed data in a 'cache' of uncompressed images. Next time when you need to load that image, look up the cache and see if there is uncompressed image there and use that. This method would work really well as you don't need to write importers/exporters anywhere, artists are happy and you are happy since you only need to decompressed each PNG file once.

Then there is the problem of where would you even want to use uncompressed data? PNG decompression on desktop computers is blazingly fast. The only place one would might want uncompressed image data is on mobile devices. The problem is that on those devices disk space is very sparse.

--EDIT
I wrote that compression ratio is 4 on average, but it actually depends on what kind of images you have. It was 4 for a game that I have. The compression ratio could be much smaller or much much bigger depending on the image that you have.
Offline ShadedVertex
« Reply #22 - Posted 2016-07-04 11:15:33 »

You guys are just mad because I have a good file format that you didn't think of.

You are conceited.

Check this link out: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

This one is pretty informative too: https://en.wikipedia.org/wiki/Illusory_superiority
Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Reply #23 - Posted 2016-07-04 17:45:45 »

This is funny.

You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Offline ShadedVertex
« Reply #24 - Posted 2016-07-05 08:13:32 »

This is funny.

It certainly is Smiley
Offline SuperMario
« Reply #25 - Posted 2016-07-05 08:48:27 »

I wouldn't like to be this guy's manager  persecutioncomplex
Offline Slyth2727
« Reply #26 - Posted 2016-07-06 03:29:48 »

I defended the wrong side. It seems that Hydroque needs to chill now :/

As a side note it is fun to read through the drama.
Offline Hydroque

JGO Coder


Medals: 25
Exp: 5 years


I'm always inspiring a good time.


« Reply #27 - Posted 2016-07-06 04:06:52 »

Let this thread die now.

You think I haven't been monitoring the chat? http://pastebin.java-gaming.org/c47d35366491fHere is a compilation <3
Offline Brynn

JGO Wizard


Medals: 103
Projects: 3
Exp: 1 month or less


JGO's Spiffy Duchess


« Reply #28 - Posted 2016-07-06 04:35:15 »

When do you plan to add audio support to the XM series of file formats?  Roll Eyes

Welcome to a new kind of tension
All across the alienation
Where everything isn't meant to be okay
Pages: [1]
  ignore  |  Print  
 
 

 
xxMrPHDxx (17 views)
2017-11-21 16:21:00

xxMrPHDxx (11 views)
2017-11-21 16:14:31

xxMrPHDxx (14 views)
2017-11-21 16:10:57

Ecumene (114 views)
2017-09-30 02:57:34

theagentd (148 views)
2017-09-26 18:23:31

cybrmynd (248 views)
2017-08-02 12:28:51

cybrmynd (247 views)
2017-08-02 12:19:43

cybrmynd (244 views)
2017-08-02 12:18:09

Sralse (258 views)
2017-07-25 17:13:48

Archive (878 views)
2017-04-27 17:45:51
List of Learning Resources
by elect
2017-03-13 14:05:44

List of Learning Resources
by elect
2017-03-13 14:04:45

SF/X Libraries
by philfrei
2017-03-02 08:45:19

SF/X Libraries
by philfrei
2017-03-02 08:44:05

SF/X Libraries
by SkyAphid
2017-03-02 06:38:56

SF/X Libraries
by SkyAphid
2017-03-02 06:38:32

SF/X Libraries
by SkyAphid
2017-03-02 06:38:05

SF/X Libraries
by SkyAphid
2017-03-02 06:37:51
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!