Java-Gaming.org Hi !
Featured games (90)
games approved by the League of Dukes
Games in Showcase (741)
Games in Android Showcase (225)
games submitted by our members
Games in WIP (823)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Tagged Human JSON (THJSON) Parser  (Read 3646 times)
0 Members and 1 Guest are viewing this topic.
Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Posted 2017-07-11 14:51:40 »

There is a git repository here:

https://github.com/Puppygames/thjson

Sooner or later, all developers try to store some sort of structured data to configure their applications in some way. Perhaps it's simple configuration files pointing to databases, perhaps it's data files for game entities, perhaps it's i18n translation files.

There are a million data formats that you might consider using. Probably the five most popular formats are:

  • INI files
  • XML files
  • YAML files
  • JSON files
  • HJSON files

Each has advantages and disadvantages for various use cases. But what if you wanted a format that comfortably handled ALL use cases?

INI files can't handle any kind of nested structure. The only concept is blocks of key=value pairs where each value must fit on a single line. It allows comments
XML files are huge and verbose, and the syntax is unnecessarily pernickety for human beings to write. It is well-specified but bloated with features that can tie you in knots. It does cope with arbitrarily complex data well, though this can be its undoing. You can comment it but often in a restricted manner. But seriously, it's a faff and everybody hates typing it.
YAML is a hideous format designed by devils. It contains a kitchen sink, manufactured in Hell. It too is pernickety with syntax requiring actually properly formatted whitespace, which is a sure sign of Satan's work. We shall not speak of it again.
JSON files are very simple to parse for a computer. Unfortunately the syntax is also unnecessarily pernickety for human beings to type and hurts the eyes somewhat with all the unnecessary quotes and commas, and the only data structures it can directly deal with are primitive values, arrays, and maps. What's worse is it can't have comments in it, which renders it pretty useless for humans.
HJSON is getting close to perfection. You can read all about HJSON at https://hjson.org and even try it out in the browser. HJSON removes all the unnecessary syntactical cruft from JSON, making mostly everything optional; and it adds a few new features such as comments in every style imaginable and multiline strings. HJSON is easily converted into JSON for feeding to other APIs. Unfortunately HJSON has one major flaw, which is that it cannot handle the concept of "classes". We can have null, numbers, booleans, strings, arrays, and maps, all nested arbitrarily... but we cannot declare a map to be a structured type, or "class". This is a big pain in the arse if you are dealing with modern OOP languages, because they deal a whole lot in classes.

Enter THJSON.

THJSON looks almost exactly like HJSON - it is in fact a superset of HJSON, which itself is a superset of JSON. The extra bit is the addition of a class name before a map (maps are objects that are enclosed in {} parentheses):
1  
2  
3  
4  
left_hand: sword {
    damage: 3
    weight: 1kg
}

Essentially any value that is parsed as a string that then starts to define a map with the opening curly brace, becomes instead a class, of a type name which is the string so far. No whitespace is allowed in the type name unless it is quoted.

We can do the same for arrays - define a class type for the array. We don't actually check that the elements themselves conform to the type - indeed we don't do anything with the class type name other than pass it to the stream listener:
1  
inventory: item [sword, axe, shoes, tea, "no tea"]

The streaming parser presented here is pretty high-speed and generates almost no garbage. In fact it only even has to allocate any objects if you start using escapes in string data. It otherwise accepts UTF-8 data as a byte[] array, which can be in either Windows, *nix or Mac line ending format, and will fire off a streamed sequence of events to a listener pointing at subsections of the input array as tokens.

In the git repository there's an example listener that converts the stream of tokens into a Google Json object. Classes are converted into JSON objects by simply creating a property called "class":
1  
2  
3  
4  
5  
"left_hand":{
    "class":"sword",
    "damage":3,
    "weight":"1kg"
}

Lists create an object of class "array" with a property "elements":
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
"inventory":{
    "class":"array",
    "elements":
        [
            "sword",
            "axe",
            "shoes",
            "tea",
            "no tea"
        ]
    }

That's not the "definitive" way to do it but it's what the example listener does. That's about it really.

Note that there isn't a complementary THJSON writer in this repository... the stream reader presented here is "lossy" in that it discards comments and whitespace. It probably wouldn't be too hard to create a writer. Maybe that's next.

Cas Smiley

Note
I cannot stand git. I do not understand git. Whatever UI comes with Eclipse these days serves only to make it impossibly complex. I have no idea how to use it and whenever I try it literally refuses to do anything, at all. In the end I resorted to dragging and dropping my src folder onto the github webpage to add files. How they could make something so simple as source control so impossible in Eclipse is a mystery. Anyway... don't ask me to do anything with this repo, because I can't figure it out.



Offline SHC
« Reply #1 - Posted 2017-07-11 14:55:15 »

Wait, a repost by mistake I guess? Seems removed soon before I post..

By the way, if you wanted to learn GIT, use the command line. I find myself lost in a different galaxy far far away when using IDE tools.

Offline klaus
« Reply #2 - Posted 2017-07-11 15:03:18 »

Wait, a repost by mistake I guess? Seems removed soon before I post..

By the way, if you wanted to learn GIT, use the command line. I find myself lost in a different galaxy far far away when using IDE tools.
Jup, I can confirm that. Learnt it by using the command line. I still do complex GIT related stuff on the command line, for the basic commit/push/pull I use the IDE.

EDIT: But to stay on-topic. Cool thing! I didn't know of HJSON and THJSON certainly looks like a neat thing to use for structured config data. For the configuration of my game entities I use spring beans. For complex configurations I think they are a good choice, because you can inherit from other configured spring beans.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #3 - Posted 2017-07-11 15:06:23 »

I couldn't even get the basic stuff to work from Eclipse with github. I literally couldn't figure it out, at all, even with the whole of the Internet to help me. I think maybe that's indicative of some sort of major problem... I was sort of expecting them to streamline it for idiots like myself after a few years but here we are a few years later and it's still ludicrous.

SVN FTW.

Cas Smiley

Offline CJC

Junior Devvie


Medals: 9
Exp: 7 years


Just press buttons until it works.


« Reply #4 - Posted 2017-07-11 16:08:41 »

I couldn't even get the basic stuff to work from Eclipse with github. I literally couldn't figure it out, at all, even with the whole of the Internet to help me. I think maybe that's indicative of some sort of major problem... I was sort of expecting them to streamline it for idiots like myself after a few years but here we are a few years later and it's still ludicrous.

SVN FTW.

Cas Smiley

IntelliJ makes it relatively easy. I'm no git expert, but like the others, I'd recommend looking at the command line, once you understand even the basics, it'll be easier to use the tools.

This would definitely be more useful with a writer! Get on it Cas! Tongue
Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #5 - Posted 2017-07-11 16:15:29 »

Hmm... exactly what input would the writer actually take?

Cas Smiley

Offline CJC

Junior Devvie


Medals: 9
Exp: 7 years


Just press buttons until it works.


« Reply #6 - Posted 2017-07-11 21:12:09 »

Good question. Not sure. Some form of JSON maybe, with the name of the maps (for instance) being surrounded by backslashes instead of quotes?
Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #7 - Posted 2017-07-11 21:15:57 »

I suppose I could just do it in reverse... provide a consumer of events that a writer will just use to build up a big ol' pretty output string.
Shouldn't take that long...

Cas Smiley

Offline nsigma
« Reply #8 - Posted 2017-07-11 21:49:33 »

SVN FTW.

For luddites everywhere  Wink

https://help.github.com/articles/support-for-subversion-clients/

Praxis LIVE - hybrid visual IDE for (live) creative coding
Offline dime26

JGO Ninja


Medals: 60
Projects: 7
Exp: 12 years


Should traffic wardens be armed?


« Reply #9 - Posted 2017-07-11 23:25:55 »

Looks interesting! Plus you made me laugh with your comments.. "YAML is a hideous format designed by devils" Cheesy

I really love Git but as someone else stated just use the command line.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #10 - Posted 2017-07-11 23:28:26 »

Nearly finished the output side of it too now... more work on it tomorrow.

Cas Smiley

Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #11 - Posted 2017-07-12 14:32:42 »

There we go, all done. You can now write THJSON as well. It can be fed either JSON, a Map<String, Object>, or POJOs.

POJO output is rather simple and comes with some specific restrictions:
1. Object graph must be acyclic!
2. There are no "references"... everything is written out verbatim, every time
3. Though you might expect it to, the POJO writer does not natively understand Lists or Maps in your POJOs because it can't make guarantees about the generic types or even the class involved. So it outputs them like it outputs any other POJO.
4. ... however it can handle arrays and other nested POJOs.

In theory I could also make an annotation-controlled Java object serializer that would address some of these shortcomings, and introduce a proper object graph that can handle references and cycles, but that's for another day.

I've not made the POJO deserializer yet... maybe tonight.

Also fixed numerous little bugs in parsing and output.

Cas Smiley

Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #12 - Posted 2017-07-18 09:32:54 »

So I rewrote the parser completely. It's now hideously complicated Cheesy But it also now works for any input I've cared to throw at it, and isn't going to blow up with a StackOverflowError if someone creates a rather arbitrarily complex thjson document.

I've also added a new feature: @directives and @functions.

At the root of the thjson document, @directives are parsed verbatim and sent to the listener, and you can then interpret the data after the @ in any way you like. For example, you could use it to do some sort of "include" system:
1  
2  
@include "/thingy.thjson"
@include http://www.thjson.org/test.thjson # Comments at end of line are not passed to listener though


You can include @functions in any place where a member value or array value is present. The contents of the @function are again passed to the listener to be interpreted in any way you like, and you must return a String which is then given to the parser as a substituted value. The value itself is then parsed to see if it's a boolean literal, number, or string:
1  
{ x: @getX, y: @getY, z: [@getZ] }


Cas Smiley

Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #13 - Posted 2017-07-19 10:28:47 »

Hmm so I think trying to be fancy and have a listener return events on a byte[] array with offsets and lengths is maybe a bit too much of a step too far in the quest for "efficiency" at the expense of "flexibility" and "utility".

Simply accepting input only in the form of a byte[] array is way less flexible than an InputStream, even if it is "faster".

Similarly, passing "strings" to the listener in the form of subsets of the original byte[] array with offsets and lengths, when world+dog is just going to turn it into a String anyway, also seems a bit onerous. And likewise the parsing of values which is an exercise rather pointlessly left to the implementation of the listener.

So I think I'm going to change it all around to use InputStream and return Strings and parsed values to the listener, super-efficiency be damned. It's more useful if it's ... useful.

Cas Smiley

Offline Riven
Administrator

« JGO Overlord »


Medals: 1324
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #14 - Posted 2017-07-19 15:27:41 »

Who's demanding this super-efficiency anyway? Get it to be 'fast enough', like everything else in a library/engine/game.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #15 - Posted 2017-07-19 16:12:50 »

Well, exactly. It will still be very bloody quick.

Cas Smiley

Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #16 - Posted 2017-07-21 10:03:44 »

Rewrote it all to be basically "user friendly" instead of "fiddly but fast". It's now layered slightly, using a special input stream, which feeds a tokenizer, which then feeds a parser, which then spits out actual parsed strings and numbers. Everything possible to slow it down but make it simple Smiley

Did some benchmarks - meaningless of course relative to everyone else but relative to each other quite interesting:

Raw "getResourceAsStream" read() throughput is 17MB/sec on this machine
The THJSONInputStream read() reads at 8MB/sec, keeping track of line & column number and converting all line endings into Unix format (single \n)
The THJSONTokenizer also manages 8MB/sec despite all the extra thinking it does to read tokens and construct them
The THJSONReader parses thjson at a rate of 7MB/sec, despite all the extra thinking it does to ensure tokens are being read in the correct order

I'm happy with reading 7MB of data in a second, I think. What is actually being done with the data that is being read is likely to be far slower.

Cas Smiley

Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #17 - Posted 2017-07-26 09:12:19 »

Latest version now committed to Github. I've included a /lib/thjson.xml Notepad++ syntax definition file that colours THJSON files reasonably well.

This version now uses the results of @function call directives and feeds them back into the parser - though the default listener interface simply returns the original @directive as a JSON string so if you don't want to do anything special then nothing special happens. If necessary you can enclose the entire call in quotes.

Example:
1  
{ person: @"Sausage McGinty" }


The string is passed to the function() method in your listener. You can parse it any way you like. You could implement it as follows:
1  
2  
3  
public String function(String text) {
   return "{ name: \""+text+"\", age: 44 }";
}


This would then be seen by the parser as:
1  
{ person: { name: "Sausage McGinty", age:44 } }


It'll allow a stack depth down to 16 levels if you get rather fancy and start having the returned thjson itself containing @calls...

So there you have it... macros.

Stuff returned from root-level @directives is not parsed - instead that's for you to control the listener in arbitrary ways. The best uses I can think of it are some sort of "#include" functionality and some way to map class name tags to actual Java class names in this sort of manner:
1  
2  
@alias monster net.puppygames.treasuretomb.Monster
@alias bullet net.puppygames.treasuretomb.Bullet


I do hope people find a use for this code because it's what I've been sorta looking for, for the last 15 years Smiley

Cas Smiley

Offline princec

« JGO Spiffy Duke »


Medals: 974
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #18 - Posted 2017-09-05 09:05:22 »

I've now added binary markup to thjson: https://github.com/Puppygames/thjson/blob/master/README.md

You can mark up binary in standard Base64 encoding either on a single line with ` quotes:
1  
2  
3  
4  
5  
6  
   base641:`YW55IGNhcm5hbCBwbGVhcw`
   base642:`YW55IGNhcm5hbCBwbGVhc3U`
   base643:`YW55IGNhcm5hbCBwbGVhc3Vy`
   base644:`YW55IGNhcm5hbCBwbGVhcw==`
   base645:`YW55IGNhcm5hbCBwbGVhc3U=`
   base646:`YW55IGNhcm5hbCBwbGVhc3Vy`

or as multiline with <<< and >>> tags:
1  
2  
3  
4  
5  
6  
   multilineBase64test:
     <<<
     abcdefghijklmnopqrstuvwxyz
     ABCDEFGHIJKLMNOPQRSTUVWXYZ
     0123456789+/
     >>>

The data are returned to the listener as byte[]. I think that's pretty much the last thing needed for this markup. Base64 is obviously not the most massively efficient binary format in the universe but it does have the advantage of being guaranteed human-readable and there's a lot of APIS out there that use Base64.

Cas Smiley

Offline Icecore
« Reply #19 - Posted 2017-09-05 10:24:48 »

We need even Moarrrrrr custom syntax and Languages  Roll Eyes
(it's not sarcasm, we really need them
 – but not stupid copy past or change 2-3 lines of exist like some kids do,
"you can do it Alone – it's not so hard... really not so hard")

Last known State: Reassembled in Cyberspace
End Transmission....
..
.
Pages: [1]
  ignore  |  Print  
 
 

 
Ecumene (110 views)
2017-09-30 02:57:34

theagentd (145 views)
2017-09-26 18:23:31

cybrmynd (245 views)
2017-08-02 12:28:51

cybrmynd (241 views)
2017-08-02 12:19:43

cybrmynd (240 views)
2017-08-02 12:18:09

Sralse (254 views)
2017-07-25 17:13:48

Archive (872 views)
2017-04-27 17:45:51

buddyBro (1016 views)
2017-04-05 03:38:00

CopyableCougar4 (1573 views)
2017-03-24 15:39:42

theagentd (1373 views)
2017-03-24 15:32:08
List of Learning Resources
by elect
2017-03-13 14:05:44

List of Learning Resources
by elect
2017-03-13 14:04:45

SF/X Libraries
by philfrei
2017-03-02 08:45:19

SF/X Libraries
by philfrei
2017-03-02 08:44:05

SF/X Libraries
by SkyAphid
2017-03-02 06:38:56

SF/X Libraries
by SkyAphid
2017-03-02 06:38:32

SF/X Libraries
by SkyAphid
2017-03-02 06:38:05

SF/X Libraries
by SkyAphid
2017-03-02 06:37:51
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!