Java 4k Resources Thread

[Eli Delventhal]

I think that there are so many different compression tools and byte-saving techniques that the common programmer doesn't know about that it can be a real detriment to them, and often a bit unfair. We've had threads like this one before, so I figured I'd start a new one.

Applet Templates
Start with a good applet template. It will have all the base functionality you need in the fewest bytes possible.
http://www.java-gaming.org/topics/applet-templates/21626/view.html

Compress your class
There are several ways you can do this. Riven has amazingly provided us with an HTTP service that does all the dirty work for you.
Compile 'n Shrink - HTTP Service - http://www.indiespot.net/app/java-four-kay

If you do not use Riven's tool, then you should use Pack200 and at least one compression tool (maybe all of them!). This can be a bit time consuming but will save you a massive number of bytes.

Pack200
No matter how you do it, you will want your game to eventually be compressed with Pack200.
http://java.sun.com/j2se/1.5.0/docs/api/java/util/jar/Pack200.html
http://java.sun.com/j2se/1.5.0/docs/guide/deployment/deployment-guide/pack200.html

Compression Tools and Obfuscators
There are lots of different tools out there to squeeze a few extra bytes out of your jar. I will eventually include specific guides with how to use each tool as they are written.
ProGuard - http://proguard.sourceforge.net/
JShrink - http://www.e-t.com/jshrink.html
JoGa - http://www.nq4.de/

Smart Techniques
Note that a lot of these things will be automatically adjusted by most optimizers. Still isn't it fun to do a lot of it manually?

• All global variables should be named only a single character. Local variables can have whatever name you want.
• All constants should be declared static final so they can go in the constants pool.
• In general, you should use the shortest possible method names you can, like System.nanoTime() versus System.currentTimeMillis().
• Every string literal you create adds at least a couple bytes per character. As a result, keep your strings short.
• Every time you use a method, it will add bytes equal to the number of characters in that method. However, each method will be added only once for all uses.
• Keep your code all in one class, and put all your code in one Applet method (like start()).
• Use local variables wherever possible. Because your game is in one method, you should be able to avoid using almost any globals.
• When concatenating numbers to strings, use String.valueOf() rather than directly adding them together. This will save many bytes.

External Links
A nice big 4k guide: http://wiki.java.net/bin/view/Games/4KGamesDesign

Please please add replies to this, especially extra tips and guides on how to use the compression tools and pack200. We want this thread to eventually be full of step-by-step guides so that people can focus on their code.

[Abuse]

I'm not sure whether the below was an intentional simplification, if so I apologise for being overly picky with my technical correctness

Every string literal you create adds exactly as many bytes as there are characters

Due to class files using a UTF-8 encoding (Modified UTF-8) to store string literals, this is not true.
Every string literal ( sl ) you create adds atleast 2 + sl.length bytes, and at most 2 + sl.length*3 bytes.

So if you ignore the 2 byte length, and limit your string to containing only characters from the first 127 positive byte values (~= western-latin alphabets), your statement is sort-of accurate - beyond that, it gets progressively more incorrect.

[pjt33]

• All global variables should be named only a single character. Local variables can have whatever name you want.
• All constants should be declared static final so they can go in the constants pool.
• Every time you use a method, it will add bytes equal to the number of characters in that method. However, each method will be added only once for all uses.

If the obfuscators aren't capable of doing trivial optimisations like these then are they not completely useless? More useful advice would be to use a single character name for the class, because obfuscators won't rename that.

• Keep your code all in one class, and put all your code in one Applet method (like start()).

I'm pretty sure that consensus from the earlier template discussion was that if you want your applet to behave correctly you have to spawn off a thread.

Games published by our own members! Check 'em out!

[appel]

Goodies link page from last year:

http://www.java-gaming.org/topics/links-tips-and-goodies/19461/view.html

[Eli Delventhal]

Oh yeah, I meant to add: correct me my mistakes! Some things (like the one byte per character for strings) were pretty much me working from memory of lsat year, I'm sure I've got plenty of little issues up there.

Thanks for the clarifications. I'll keep editing this as we get more info. I have no experience with the optimizers, so if some things are pointless then I won't really know it.

I'm personally annoyed I have to use optimizers to be able to contend - the fun part for me is putting in static final etc. etc. in order to get the bytes down. And then doing a smart thing with a loop, etc. When your code literally goes from 4kb to 1/2kb with optimizers and compression, that's kind of frustrating. Part of the reason I made this thread is because last year I was a bit overwhelmed with the incredibly large number of things I had to do to compress my game, none of which had anything to do with code. I think it's a really steep hill you have to climb as someone new to 4k.

[Markus_Persson]

Pack200 brings my current project down to 1.47 kb from 2.05 kb. That's very very good.

[Riven]

If somebody has the tools that run on linux, I can make my server happy by doing some bruteforce zip-attacks on your cute JARs.

Last year we such an app, but had some reliance on *.exe file, and it was macroing some GUI, and my server is blind. (who did it? Appel??)

So if anybody it willing to give it a gentle kick towards linux, I'm going to write some HTTP service that does all the naughty stuff for you.

[appel]

If somebody has the tools that run on linux, I can make my server happy by doing some bruteforce zip-attacks on your cute JARs.

Last year we such an app, but had some reliance on *.exe file, and it was macroing some GUI, and my server is blind. (who did it? Appel??)

So if anybody it willing to give it a gentle kick towards linux, I'm going to write some HTTP service that does all the naughty stuff for you.

I don't understand. Any security threat I need to worry about?

[Riven]

I don't understand. Any security threat I need to worry about?

Uh... somebody (you?) had make this massive script, that fed the JAR through at least 10 compressors/obfuscators. What I was trying to say was that I'd like to make this an online service for everybody to use. But it has to run on Linux, and without a desktop.

[Markus_Persson]

That would be really good!

Games published by our own members! Check 'em out!

[moogie]

If somebody has the tools that run on linux, I can make my server happy by doing some bruteforce zip-attacks on your cute JARs.

Last year we such an app, but had some reliance on *.exe file, and it was macroing some GUI, and my server is blind. (who did it? Appel??)

So if anybody it willing to give it a gentle kick towards linux, I'm going to write some HTTP service that does all the naughty stuff for you.

I believe that is the 4kjo tool i had made... it was not ever really robust but more of an in house tool that i shared... warts and all

http://www.java-gaming.org/topics/4kjo-4k-java-optimiser-version-3-released/18085/view.html

[moogie]

Pack200 brings my current project down to 1.47 kb from 2.05 kb. That's very very good.

hmm now i am a little confused... i dont see a way to submit a pack200 version (pack.gz) of the jar on the java4k website.

Does this mean that the java4k website is not configured to deliver pack200 compressed jars?

Or do we have to embed a pack200 version attempt to decompress and add the embedded pack200 jar to the class loader of the applet at run time?

[pjt33]

I'm personally annoyed I have to use optimizers to be able to contend - the fun part for me is putting in static final etc. etc. in order to get the bytes down. And then doing a smart thing with a loop, etc.

Last year I used Proguard and then decompiled it and saved the last 60-odd bytes by hand-tweaking the bytecode, reassembling with Jasmin. You can still be hardcore if you want to be!

[appel]

hmm now i am a little confused... i dont see a way to submit a pack200 version (pack.gz) of the jar on the java4k website.

Does this mean that the java4k website is not configured to deliver pack200 compressed jars?

Or do we have to embed a pack200 version attempt to decompress and add the embedded pack200 jar to the class loader of the applet at run time?

You can upload both .jar and .gz files as "applet jars".

[Riven]

Interresting results -- assuming they both sqeezed their JARs, as I couldn't find a tool to do it better...


Falcon4k (Alan_W)

   4,093 bytes (jar)
   3,485 bytes (pack.gz)

   4,125 bytes (re-zipped with winrar, for baseline)
   3,485 bytes (pack.gz => after winrar did the re-zip)

   3,467 bytes (pack.gz => 7z on .pack)



Flux4k (Michael Bliem)

   4,093 bytes (jar)
   3,331 bytes (pack.gz)

   3,990 bytes (jar => re-zipped with winrar, for baseline)
   3,288 bytes (pack.gz => after winrar did the re-zip)

   3,260 bytes (pack.gz => 7z on .pack)





Interesting to see that pack200 did a better job after a 'plain zipper' like WinRAR.

[Abuse]

Interresting results -- assuming they both sqeezed their JARs, as I couldn't find a tool to do it better...


Falcon4k (Alan_W)
   4,093 bytes (jar)
   4,125 bytes (re-zipped with winrar, for baseline)
   3,485 bytes (pack.gz)

Flux4k (Michael Bliem)
   4,093 bytes (jar)
   3,990 bytes (re-zipped with winrar, for baseline)
   3,331 bytes (pack.gz)

Use kzip or 7zip for the comparison against jar.
Still impressive savings though; anyone fancy writing an optimised pack200 compressor?

Nvm, redundant - you can skip the gzip step and use whatever gzip compatible super-efficient compressor you like.

I suppose the "pack"ing step might be further optimisable with a home-grown implementation.

[appel]

Perhaps not everyone are aware that they can use pack200.

[Riven]

I updated the stats, BTW...

Still impressive savings though; anyone fancy writing an optimised pack200 compressor?

pack.gz is simply Java's default GZ deflater over a .pack file.

I think that's the only point where we can optimize by doing some bruteforce GZ-ing, as pack200 has to unzip your carefully crafted jar anyway.

[Abuse]

I'm curious; does pack200 invalidate the 'inject binary data into class files' space saving paradigm?

[Riven]

I updated the table again... it's getting smaller without any effort.

[moogie]

I'm curious; does pack200 invalidate the 'inject binary data into class files' space saving paradigm?

Yes, i beleive that is the case as and class with an unknown attribuite (i.e. the binary data) is not compressed by pack 200.

however it is likely that having the data as a seperate file in a jar that is pack200 compressed will still be much smaller than an embedded data in a normal jar

[Riven]

Compile 'n Shrink - HTTP Service

http://www.indiespot.net/app/java-four-kay

[pjt33]

I used kzip last year, but it doesn't support the gzip file format. Therefore here is a utility I've knocked up which converts zip files into gzip files. It makes various assumptions (only the first file in the zip is processed; the zip is assumed to give CRC, compressed size, and uncompressed size before the data block rather than after it). I've tested it with kzip and ensured that the result is correct.

Error handling is about what you'd expect from a throwaway tool (i.e. minimal).

Not yet tested on a .pack file, but I'm about to go to bed so I'll put it up for people to play with.

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80   81   82   83   84   85   86   87   88   89   90   91   92

import java.io.*; // Based on http://www.gzip.org/zlib/rfc-gzip.html // and http://www.pkware.com/documents/casestudies/APPNOTE.TXT public class Zip2Gzip {    // Usage: java Zip2Gzip [file.zip [file.gzip]]    public static void main(String[] args) throws IOException    {       InputStream in = args.length < 1 ? System.in : new FileInputStream(args[0]);       in = new BufferedInputStream(in);       OutputStream out = args.length < 2 ? System.out : new FileOutputStream(args[1]);       out = new BufferedOutputStream(out);       // Start by writing a gzip header.       // Magic.       out.write(0x1f);       out.write(0x8b);       // Compression method: inflate.       out.write(0x08);       // Flags: we include no optional extras.       out.write(0x00);       // Timestamp: unavailable.       out.write(0x00);       out.write(0x00);       out.write(0x00);       out.write(0x00);       // Extra flags: none.       out.write(0x00);       // OS: unknown.       out.write(0xff);       // The next block of output is the compressed data. We need to process       // the zip file header to find it and know how long it is.       // local file header signature     4 bytes  (0x04034b50)       // version needed to extract       2 bytes       // general purpose bit flag        2 bytes       // compression method              2 bytes       // last mod file time              2 bytes       // last mod file date              2 bytes       //   Total so far:                14 bytes       for (int i = 0; i < 14; i++) in.read();       // crc-32                          4 bytes       int crc1 = in.read();       int crc2 = in.read();       int crc3 = in.read();       int crc4 = in.read();       // compressed size                 4 bytes       int cmpSz =  (in.read() & 0xff) +                   ((in.read() & 0xff) << 8) +                   ((in.read() & 0xff) << 16) +                   ((in.read() & 0xff) << 24);       // uncompressed size               4 bytes       int ucmpSz1 = in.read();       int ucmpSz2 = in.read();       int ucmpSz3 = in.read();       int ucmpSz4 = in.read();       // file name length                2 bytes       int nameLen = (in.read() & 0xff) + ((in.read() & 0xff) << 8);       // extra field length              2 bytes       int xfLen = (in.read() & 0xff) + ((in.read() & 0xff) << 8);       // file name (variable size)       for (int i = 0; i < nameLen; i++) in.read();       // extra field (variable size)       for (int i = 0; i < xfLen; i++) in.read();       // Data follows, so we can copy it to the output.       byte[] buf = new byte[4096];       while (cmpSz > 0)       {          int desired = cmpSz > buf.length ? buf.length : cmpSz;          int len = in.read(buf, 0, desired);          if (len == 0) throw new EOFException();          out.write(buf, 0, len);          cmpSz -= len;       }       // The output still needs the CRC32 and the uncompressed size.       out.write(crc1);       out.write(crc2);       out.write(crc3);       out.write(crc4);       out.write(ucmpSz1);       out.write(ucmpSz2);       out.write(ucmpSz3);       out.write(ucmpSz4);       // Done. Be tidy.       out.close();       in.close();    } }

[appel]

Compile 'n Shrink - HTTP Service

http://www.indiespot.net/app/java-four-kay

Cooooool!

[Riven]

I used kzip last year, but it doesn't support the gzip file format. Therefore here is a utility ...

I'm using 7z's gz to compress the .pack file. Tomorrow I will try to see how it compares to the gz part of kzip.

[moogie]

I've knocked up which converts zip files into gzip files.

Handy thanks!

[SimonH]

Compile 'n Shrink - HTTP Service

http://www.indiespot.net/app/java-four-kay

Uber-cool! Nice one!

[Alan_W]

Interresting results -- assuming they both sqeezed their JARs, as I couldn't find a tool to do it better...


Falcon4k (Alan_W)

   4,093 bytes (jar)
...
   3,467 bytes (pack.gz => 7z on .pack)

...


Interesting to see that pack200 did a better job after a 'plain zipper' like WinRAR.

Good grief.  Absolutely amazing. Thanks Riven. I knew that pack200 zipped up all the files in an archive together, thus running the compression algorithm over the lot, rather than each one individually and had therefore only expected a noticeable benefit for archives containing multiple files.  I guess pack200 must also reduce some of the class overhead that comes with a standard java class.  I had intended to target jre1.4 again this year as the only thing I wanted out of jre1.5 was the nanosecond timer, but am now sitting down with a dazed expression.  Maybe leave Falcon4k as jre1.4 as it's pretty much finished, but on the other hand, I could fit a lot of level data in that 500 extra bytes. 

[zammbi]

I say just ignore Java 1.4, or you could always make 2 versions but just submit the Java 1.5 version.

[pjt33]

I guess pack200 must also reduce some of the class overhead that comes with a standard java class.

IIRC at a minimum it exploits redundancy in the constant pool structure.