Hi !
Featured games (91)
games approved by the League of Dukes
Games in Showcase (757)
Games in Android Showcase (229)
games submitted by our members
Games in WIP (844)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Thinking about data formats - would appreciate opinions.  (Read 2166 times)
0 Members and 1 Guest are viewing this topic.
Offline phunni

Senior Newbie

Exp: 17 years

« Posted 2018-04-09 13:41:41 »

I'm getting ready to start creating some of the basic data for my game - enemies, items the player can use/equip etc. and I'm trying to decide on the best format for this data.

I know that JSON is really popular these days and it has clear advantages:

1) Fast processing
2) decent out of the box support in libGDX

but there are also things I don't like about it.  Despite what everyone on the internet seems to think, I don't find it as easy to read as XML - or, at least, well-designed XML.  XML also seems to have much better support for metadata than JSON does.  I know XML is verbose, but I genuinely don't care (I've never really understood the strong aversion to verbosity in XML or in Java) - unless someone knows of a really good technical reason to go with a less verbose format.

XML also has XPath.  The ability to perform a query on the data is important to me and I know that XPath is good for this and, I believe, can be done reasonably quickly via a cheeky SAX parse.  There is, of course, JSONPath - is that any good?

The other option is to use SQLite, which has all of the above advantages (clean structure, metadata support and queryable), but the downside of a performance and memory overhead. It's a nice technology, though, but I suspect that either XML or JSON would perform better.

Anyway, I'm interested in the thoughts of the internet - so please weigh in.  I'm expecting a lot of support for JSON, which is fine - it may well be the best solution, although please don't just recommend it because it's the most popular.  I've found a number of discussions online about this where that seems to be the main argument for JSON - just that it's the latest and greatest thing.

Obviously, if you're aware of yet another solution I'd be really interested to hear about it.
Offline princec

« JGO Spiffy Duke »

Medals: 1033
Projects: 3
Exp: 20 years

Eh? Who? What? ... Me?

« Reply #1 - Posted 2018-04-09 14:11:39 »

I'm quite fond of my THJSON parser Smiley Most of the advantages of XML, most of the advantages of JSON.

Cas Smiley

Offline phunni

Senior Newbie

Exp: 17 years

« Reply #2 - Posted 2018-04-09 14:15:02 »

Thanks - I'll take a look!

Edit: Glad to see someone else who finds JSON "hurts the eyes"
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline KevinWorkman

« JGO Plugged Duke »

Medals: 283
Projects: 12
Exp: 12 years - Coding Tutorials!

« Reply #3 - Posted 2018-04-09 15:33:48 »

If you like XML, then use XML.

How often is your game going to be reading these files? How large are they? How often do you, as a human, need to read or edit them?

My guess is they aren't going to be huge, and you aren't going to be reading them (either via code or as a human) very often, so you aren't going to notice a huge difference in "performance" no matter what you choose.

So go with whatever you like the best. - Coding Tutorials!
Happy Coding forum - Come say hello!
Offline phunni

Senior Newbie

Exp: 17 years

« Reply #4 - Posted 2018-04-09 15:43:44 »

They won't be all that large and they should only be read at the beginning of each level during initialisation.
Offline ags1

JGO Kernel

Medals: 367
Projects: 7

Make code not war!

« Reply #5 - Posted 2018-04-09 19:25:39 »

Not that I am saying it is finished yet, but if you like markup languages have a look at my TreeML, more compact than JSON and it has a nifty little schema language and a very basic path language (currently only really used in my unit tests).

Offline ByerN
« Reply #6 - Posted 2018-04-09 20:01:18 »

Well... It all depends on your intention. Tell me more about what you want to do with it.

How often do you want to get data from your filesystem? For what? Do you want to do high freq queries on your storage? Do you want to modify your storage data at runtime?

For simple configuration files I prefer json or yaml (simplicity)
For sending data over the wire in human-friendly format I prefer JSON (ofc not if we want good performance Smiley)
For configuration/data in interop with external software it's possible that xml would be better (it has advantage for transforms and sticking to scheme)
For querying, I would use relational database
No-sql would be also good idea, but it's tricky (depends what you need)
Maybe CSV?

About JSON vs XML -> I've found working with JSON much faster and simplier than XML, but sometimes you just can't do this or it's better to use xml. JSON is a popular serialization format for sending messages over the wire and I think that we know why Cheesy

Offline philfrei
« Reply #7 - Posted 2018-04-10 16:55:32 »

My view is pretty simplistic. I don't have to load any new libraries in order to use XML. Also, my programs are not pushing any envelopes (at most, just trying to ensure sufficient efficiencies for good audio processing).

I would think JSON would make the most sense in scenarios where one is either interacting with the JavaScript world or sending/receiving information from non-local URLs. If it is well integrated into LibGDX, that could also be a good reason to go for it, since overhead for handling the code is there anyway.

I agree it is nice to have the data that is saved/loaded in an easily readable form.

music and music apps:
Offline Damocles
« Reply #8 - Posted 2018-04-12 10:20:47 »

It really depends an the data regarding:
-who is editing / generating the data
-how often is the data read (loading in bulk to memory, or regularly querying it)
-is the data used to serialize state/objects and send it to another system over the network
-how important is it to be able to manually edit the data (favoring thus any text-based format)
-how much data it there (a small config file vs a JGO Database Dump)
-how many content providers have to agree on a format (making validation / schema necessary)
-how important is it to be able to manually read the data for debugging purposes (using a more verbose format early in development)

One example, that I use for processing numeric data, such as balancing data (Serverside):

I use Excel/OpenOffice and edit the data in a spreadsheet.
then I save it to an ods file (the native OpenOffice format)
This is basically just a zip file containing all the spreadsheed data, and (for my purpose unimportant) meta and format data
Then I just directly load the spreadsheet in Java, unpack it, get the "content.xml", use the default DocumentBuilder to parse it, and extract my data as needed

The advantage it, that I can directly edit the data in Excel or OpenOffice, and just hit save (confusingly being a floppy disk button)
Voila, the data can be loaded directly.

(The easy implementation here is to just parse a csv, but I have often forgot to save to the original spreadsheet afterwards, so its less fail-proof)

Editing XML or even JSON can get tiresome when using a lot of numeric fields. Here a spreadsheet is a way better presentation.
Plus, I can apply logic (formulas) and evaluation directly on the data, while editing it.


When creating level data (topography, eg, a matrix of a lot of regular data, in a few layers) I whould use a simple CSV, or compressed binary data, when it gets too big.
When creating level data, that contains a list of many objects, each having a deeper hierarchy of sub-objects and their attributes, I would use XML (JSON would be an option, but has some drawbacks here)
When creating state-data, that has to be send over the network, I would use something like JSON at first, but then move to create a custom serialization later to reduce the total size.

SQL Databases:
I would not use a SQL database unless:
-there is a LOT of data, that should not be kept in memory
-there is a need to QUERY the data, and not just read all data in a big chunk
-serialization would be too complicated to do it with a trivial data-dump to a binary file
Offline phunni

Senior Newbie

Exp: 17 years

« Reply #9 - Posted 2018-04-12 13:47:04 »

OK, to give a little more context.

The data is going to be used at the beginning of every level and only at the beginning.  The game is based on a proc gen dungeon and I want to have some sort of data file(s) to list all the possible mobs and possible items.

It absolutely does need to be queryable and that's actually causing the sticking point. In XML, you can fairly easily run an XPath query on a DOM, but that requires having a DOM which not only uses memory but is slow(ish) to create.  Alternatively, I can use a SAX parse with a subset of XPath (adequate) and gain performance, but having to use a 3rd party library - defeating one of the (admittedly minor) benefits of XML.  There is JSONPath, but, again, it requires 3rd party tools.

For example, each mob has a minimum depth attribute. So, if a mob has a MD of 10 then it mustn't appear in levels 1 - 9. So, at the beginning of each level, I'm going to do a query along the lines of "get me all the mobs with a minDepth >= x"

Hope that adds some clarity!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline princec

« JGO Spiffy Duke »

Medals: 1033
Projects: 3
Exp: 20 years

Eh? Who? What? ... Me?

« Reply #10 - Posted 2018-04-12 15:29:56 »

Reckon your best bet is to a) use the data simply as persistence, not as a database - load it into the heap in a useful queryable POJO structure and b) use thjson which is precisely designed for this exact task you had in mind Tongue

Cas Smiley

Offline phunni

Senior Newbie

Exp: 17 years

« Reply #11 - Posted 2018-04-12 17:24:52 »

I'll have a play around with THJSON and see if I can figure out how to use the classes (no docs ;-p) - by the time I've written a couple of simple test classes, it should be obvious if it's the solution for me.
Pages: [1]
  ignore  |  Print  

EgonOlsen (79 views)
2018-06-10 19:43:48

EgonOlsen (59 views)
2018-06-10 19:43:44

EgonOlsen (78 views)
2018-06-10 19:43:20

DesertCoockie (261 views)
2018-05-13 18:23:11

nelsongames (159 views)
2018-04-24 18:15:36

nelsongames (158 views)
2018-04-24 18:14:32

ivj94 (901 views)
2018-03-24 14:47:39

ivj94 (162 views)
2018-03-24 14:46:31

ivj94 (813 views)
2018-03-24 14:43:53

Solater (177 views)
2018-03-17 05:04:08
Java Gaming Resources
by philfrei
2017-12-05 19:38:37

Java Gaming Resources
by philfrei
2017-12-05 19:37:39

Java Gaming Resources
by philfrei
2017-12-05 19:36:10

Java Gaming Resources
by philfrei
2017-12-05 19:33:10

List of Learning Resources
by elect
2017-03-13 14:05:44

List of Learning Resources
by elect
2017-03-13 14:04:45

SF/X Libraries
by philfrei
2017-03-02 08:45:19

SF/X Libraries
by philfrei
2017-03-02 08:44:05 is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!