Eli Delventhal
|
 |
«
Posted
2009-12-01 16:28:12 » |
|
I think that there are so many different compression tools and byte-saving techniques that the common programmer doesn't know about that it can be a real detriment to them, and often a bit unfair. We've had threads like this one before, so I figured I'd start a new one. Applet TemplatesStart with a good applet template. It will have all the base functionality you need in the fewest bytes possible. http://www.java-gaming.org/topics/applet-templates/21626/view.htmlCompress your classThere are several ways you can do this. Riven has amazingly provided us with an HTTP service that does all the dirty work for you. Compile 'n Shrink - HTTP Service - http://www.indiespot.net/app/java-four-kayIf you do not use Riven's tool, then you should use Pack200 and at least one compression tool (maybe all of them!). This can be a bit time consuming but will save you a massive number of bytes. Pack200No matter how you do it, you will want your game to eventually be compressed with Pack200. http://java.sun.com/j2se/1.5.0/docs/api/java/util/jar/Pack200.htmlhttp://java.sun.com/j2se/1.5.0/docs/guide/deployment/deployment-guide/pack200.htmlCompression Tools and ObfuscatorsThere are lots of different tools out there to squeeze a few extra bytes out of your jar. I will eventually include specific guides with how to use each tool as they are written. ProGuard - http://proguard.sourceforge.net/JShrink - http://www.e-t.com/jshrink.htmlJoGa - http://www.nq4.de/Smart TechniquesNote that a lot of these things will be automatically adjusted by most optimizers. Still isn't it fun to do a lot of it manually?  - All global variables should be named only a single character. Local variables can have whatever name you want.
- All constants should be declared static final so they can go in the constants pool.
- In general, you should use the shortest possible method names you can, like System.nanoTime() versus System.currentTimeMillis().
- Every string literal you create adds at least a couple bytes per character. As a result, keep your strings short.
- Every time you use a method, it will add bytes equal to the number of characters in that method. However, each method will be added only once for all uses.
- Keep your code all in one class, and put all your code in one Applet method (like start()).
- Use local variables wherever possible. Because your game is in one method, you should be able to avoid using almost any globals.
- When concatenating numbers to strings, use String.valueOf() rather than directly adding them together. This will save many bytes.
External LinksA nice big 4k guide: http://wiki.java.net/bin/view/Games/4KGamesDesignPlease please add replies to this, especially extra tips and guides on how to use the compression tools and pack200. We want this thread to eventually be full of step-by-step guides so that people can focus on their code.
|
|
|
|
Abuse
|
 |
«
Reply #1 - Posted
2009-12-01 17:21:05 » |
|
I'm not sure whether the below was an intentional simplification, if so I apologise for being overly picky with my technical correctness  Every string literal you create adds exactly as many bytes as there are characters Due to class files using a UTF-8 encoding ( Modified UTF-8) to store string literals, this is not true. Every string literal ( sl ) you create adds atleast 2 + sl.length bytes, and at most 2 + sl.length*3 bytes. So if you ignore the 2 byte length, and limit your string to containing only characters from the first 127 positive byte values (~= western-latin alphabets), your statement is sort-of accurate - beyond that, it gets progressively more incorrect.
|
|
|
|
pjt33
|
 |
«
Reply #2 - Posted
2009-12-01 17:28:13 » |
|
- All global variables should be named only a single character. Local variables can have whatever name you want.
- All constants should be declared static final so they can go in the constants pool.
- Every time you use a method, it will add bytes equal to the number of characters in that method. However, each method will be added only once for all uses.
If the obfuscators aren't capable of doing trivial optimisations like these then are they not completely useless? More useful advice would be to use a single character name for the class, because obfuscators won't rename that. - Keep your code all in one class, and put all your code in one Applet method (like start()).
I'm pretty sure that consensus from the earlier template discussion was that if you want your applet to behave correctly you have to spawn off a thread.
|
|
|
|
Games published by our own members! Check 'em out!
|
|
appel
|
 |
«
Reply #3 - Posted
2009-12-01 17:33:17 » |
|
|
|
|
|
Eli Delventhal
|
 |
«
Reply #4 - Posted
2009-12-01 19:57:43 » |
|
Oh yeah, I meant to add: correct me my mistakes! Some things (like the one byte per character for strings) were pretty much me working from memory of lsat year, I'm sure I've got plenty of little issues up there.
Thanks for the clarifications. I'll keep editing this as we get more info. I have no experience with the optimizers, so if some things are pointless then I won't really know it.
I'm personally annoyed I have to use optimizers to be able to contend - the fun part for me is putting in static final etc. etc. in order to get the bytes down. And then doing a smart thing with a loop, etc. When your code literally goes from 4kb to 1/2kb with optimizers and compression, that's kind of frustrating. Part of the reason I made this thread is because last year I was a bit overwhelmed with the incredibly large number of things I had to do to compress my game, none of which had anything to do with code. I think it's a really steep hill you have to climb as someone new to 4k.
|
|
|
|
Markus_Persson
|
 |
«
Reply #5 - Posted
2009-12-01 20:52:47 » |
|
Pack200 brings my current project down to 1.47 kb from 2.05 kb. That's very very good. 
|
|
|
|
Riven
|
 |
«
Reply #6 - Posted
2009-12-01 20:53:17 » |
|
If somebody has the tools that run on linux, I can make my server happy by doing some bruteforce zip-attacks on your cute JARs.
Last year we such an app, but had some reliance on *.exe file, and it was macroing some GUI, and my server is blind. (who did it? Appel??)
So if anybody it willing to give it a gentle kick towards linux, I'm going to write some HTTP service that does all the naughty stuff for you.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings!
|
|
|
appel
|
 |
«
Reply #7 - Posted
2009-12-01 20:58:52 » |
|
If somebody has the tools that run on linux, I can make my server happy by doing some bruteforce zip-attacks on your cute JARs.
Last year we such an app, but had some reliance on *.exe file, and it was macroing some GUI, and my server is blind. (who did it? Appel??)
So if anybody it willing to give it a gentle kick towards linux, I'm going to write some HTTP service that does all the naughty stuff for you.
I don't understand. Any security threat I need to worry about?
|
|
|
|
Riven
|
 |
«
Reply #8 - Posted
2009-12-01 21:01:59 » |
|
I don't understand. Any security threat I need to worry about?
Uh... somebody (you?) had make this massive script, that fed the JAR through at least 10 compressors/obfuscators. What I was trying to say was that I'd like to make this an online service for everybody to use. But it has to run on Linux, and without a desktop.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings!
|
|
|
Markus_Persson
|
 |
«
Reply #9 - Posted
2009-12-01 21:11:37 » |
|
That would be really good!
|
|
|
|
Games published by our own members! Check 'em out!
|
|
moogie
|
 |
«
Reply #10 - Posted
2009-12-01 21:19:11 » |
|
If somebody has the tools that run on linux, I can make my server happy by doing some bruteforce zip-attacks on your cute JARs.
Last year we such an app, but had some reliance on *.exe file, and it was macroing some GUI, and my server is blind. (who did it? Appel??)
So if anybody it willing to give it a gentle kick towards linux, I'm going to write some HTTP service that does all the naughty stuff for you.
I believe that is the 4kjo tool i had made... it was not ever really robust  but more of an in house tool that i shared... warts and all http://www.java-gaming.org/topics/4kjo-4k-java-optimiser-version-3-released/18085/view.html
|
Java4k RIP 2014
|
|
|
moogie
|
 |
«
Reply #11 - Posted
2009-12-01 21:22:20 » |
|
Pack200 brings my current project down to 1.47 kb from 2.05 kb. That's very very good.  hmm now i am a little confused... i dont see a way to submit a pack200 version (pack.gz) of the jar on the java4k website. Does this mean that the java4k website is not configured to deliver pack200 compressed jars? Or do we have to embed a pack200 version attempt to decompress and add the embedded pack200 jar to the class loader of the applet at run time?
|
Java4k RIP 2014
|
|
|
pjt33
|
 |
«
Reply #12 - Posted
2009-12-01 21:34:36 » |
|
I'm personally annoyed I have to use optimizers to be able to contend - the fun part for me is putting in static final etc. etc. in order to get the bytes down. And then doing a smart thing with a loop, etc.
Last year I used Proguard and then decompiled it and saved the last 60-odd bytes by hand-tweaking the bytecode, reassembling with Jasmin. You can still be hardcore if you want to be!
|
|
|
|
appel
|
 |
«
Reply #13 - Posted
2009-12-01 21:35:38 » |
|
hmm now i am a little confused... i dont see a way to submit a pack200 version (pack.gz) of the jar on the java4k website.
Does this mean that the java4k website is not configured to deliver pack200 compressed jars?
Or do we have to embed a pack200 version attempt to decompress and add the embedded pack200 jar to the class loader of the applet at run time?
You can upload both .jar and .gz files as "applet jars".
|
|
|
|
Riven
|
 |
«
Reply #14 - Posted
2009-12-01 21:49:29 » |
|
Interresting results -- assuming they both sqeezed their JARs, as I couldn't find a tool to do it better...
Falcon4k (Alan_W)
4,093 bytes (jar) 3,485 bytes (pack.gz)
4,125 bytes (re-zipped with winrar, for baseline) 3,485 bytes (pack.gz => after winrar did the re-zip)
3,467 bytes (pack.gz => 7z on .pack)
Flux4k (Michael Bliem)
4,093 bytes (jar) 3,331 bytes (pack.gz)
3,990 bytes (jar => re-zipped with winrar, for baseline) 3,288 bytes (pack.gz => after winrar did the re-zip)
3,260 bytes (pack.gz => 7z on .pack)
Interesting to see that pack200 did a better job after a 'plain zipper' like WinRAR.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings!
|
|
|
Abuse
|
 |
«
Reply #15 - Posted
2009-12-01 21:55:19 » |
|
Interresting results -- assuming they both sqeezed their JARs, as I couldn't find a tool to do it better...
Falcon4k (Alan_W) 4,093 bytes (jar) 4,125 bytes (re-zipped with winrar, for baseline) 3,485 bytes (pack.gz)
Flux4k (Michael Bliem) 4,093 bytes (jar) 3,990 bytes (re-zipped with winrar, for baseline) 3,331 bytes (pack.gz)
Use kzip or 7zip for the comparison against jar. Still impressive savings though; anyone fancy writing an optimised pack200 compressor?Nvm, redundant - you can skip the gzip step and use whatever gzip compatible super-efficient compressor you like. I suppose the "pack"ing step might be further optimisable with a home-grown implementation.
|
|
|
|
appel
|
 |
«
Reply #16 - Posted
2009-12-01 21:57:11 » |
|
Perhaps not everyone are aware that they can use pack200.
|
|
|
|
Riven
|
 |
«
Reply #17 - Posted
2009-12-01 21:59:55 » |
|
I updated the stats, BTW... Still impressive savings though; anyone fancy writing an optimised pack200 compressor?
pack.gz is simply Java's default GZ deflater over a .pack file. I think that's the only point where we can optimize by doing some bruteforce GZ-ing, as pack200 has to unzip your carefully crafted jar anyway.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings!
|
|
|
Abuse
|
 |
«
Reply #18 - Posted
2009-12-01 22:11:03 » |
|
I'm curious; does pack200 invalidate the 'inject binary data into class files' space saving paradigm?
|
|
|
|
Riven
|
 |
«
Reply #19 - Posted
2009-12-01 22:14:03 » |
|
I updated the table again... it's getting smaller without any effort.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings!
|
|
|
moogie
|
 |
«
Reply #20 - Posted
2009-12-01 22:55:04 » |
|
I'm curious; does pack200 invalidate the 'inject binary data into class files' space saving paradigm?
Yes, i beleive that is the case as and class with an unknown attribuite (i.e. the binary data) is not compressed by pack 200. however it is likely that having the data as a seperate file in a jar that is pack200 compressed will still be much smaller than an embedded data in a normal jar
|
Java4k RIP 2014
|
|
|
Riven
|
 |
«
Reply #21 - Posted
2009-12-01 23:22:23 » |
|
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings!
|
|
|
pjt33
|
 |
«
Reply #22 - Posted
2009-12-02 00:14:56 » |
|
I used kzip last year, but it doesn't support the gzip file format. Therefore here is a utility I've knocked up which converts zip files into gzip files. It makes various assumptions (only the first file in the zip is processed; the zip is assumed to give CRC, compressed size, and uncompressed size before the data block rather than after it). I've tested it with kzip and ensured that the result is correct. Error handling is about what you'd expect from a throwaway tool (i.e. minimal). Not yet tested on a .pack file, but I'm about to go to bed so I'll put it up for people to play with. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
| import java.io.*;
public class Zip2Gzip { public static void main(String[] args) throws IOException { InputStream in = args.length < 1 ? System.in : new FileInputStream(args[0]); in = new BufferedInputStream(in); OutputStream out = args.length < 2 ? System.out : new FileOutputStream(args[1]); out = new BufferedOutputStream(out);
out.write(0x1f); out.write(0x8b); out.write(0x08); out.write(0x00); out.write(0x00); out.write(0x00); out.write(0x00); out.write(0x00); out.write(0x00); out.write(0xff);
for (int i = 0; i < 14; i++) in.read(); int crc1 = in.read(); int crc2 = in.read(); int crc3 = in.read(); int crc4 = in.read(); int cmpSz = (in.read() & 0xff) + ((in.read() & 0xff) << 8) + ((in.read() & 0xff) << 16) + ((in.read() & 0xff) << 24); int ucmpSz1 = in.read(); int ucmpSz2 = in.read(); int ucmpSz3 = in.read(); int ucmpSz4 = in.read(); int nameLen = (in.read() & 0xff) + ((in.read() & 0xff) << 8); int xfLen = (in.read() & 0xff) + ((in.read() & 0xff) << 8); for (int i = 0; i < nameLen; i++) in.read(); for (int i = 0; i < xfLen; i++) in.read();
byte[] buf = new byte[4096]; while (cmpSz > 0) { int desired = cmpSz > buf.length ? buf.length : cmpSz; int len = in.read(buf, 0, desired); if (len == 0) throw new EOFException(); out.write(buf, 0, len); cmpSz -= len; }
out.write(crc1); out.write(crc2); out.write(crc3); out.write(crc4); out.write(ucmpSz1); out.write(ucmpSz2); out.write(ucmpSz3); out.write(ucmpSz4);
out.close(); in.close(); } } |
|
|
|
|
appel
|
 |
«
Reply #23 - Posted
2009-12-02 00:56:39 » |
|
Cooooool! 
|
|
|
|
Riven
|
 |
«
Reply #24 - Posted
2009-12-02 01:24:23 » |
|
I used kzip last year, but it doesn't support the gzip file format. Therefore here is a utility ...
I'm using 7z's gz to compress the .pack file. Tomorrow I will try to see how it compares to the gz part of kzip.
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings!
|
|
|
moogie
|
 |
«
Reply #25 - Posted
2009-12-02 02:04:55 » |
|
I've knocked up which converts zip files into gzip files. Handy  thanks!
|
Java4k RIP 2014
|
|
|
SimonH
|
 |
«
Reply #26 - Posted
2009-12-02 02:41:21 » |
|
|
People make games and games make people
|
|
|
Alan_W
|
 |
«
Reply #27 - Posted
2009-12-02 05:33:45 » |
|
Interresting results -- assuming they both sqeezed their JARs, as I couldn't find a tool to do it better...
Falcon4k (Alan_W)
4,093 bytes (jar) ... 3,467 bytes (pack.gz => 7z on .pack)
...
Interesting to see that pack200 did a better job after a 'plain zipper' like WinRAR.
Good grief. Absolutely amazing. Thanks Riven. I knew that pack200 zipped up all the files in an archive together, thus running the compression algorithm over the lot, rather than each one individually and had therefore only expected a noticeable benefit for archives containing multiple files. I guess pack200 must also reduce some of the class overhead that comes with a standard java class. I had intended to target jre1.4 again this year as the only thing I wanted out of jre1.5 was the nanosecond timer, but am now sitting down with a dazed expression. Maybe leave Falcon4k as jre1.4 as it's pretty much finished, but on the other hand, I could fit a lot of level data in that 500 extra bytes. 
|
Time flies like a bird. Fruit flies like a banana.
|
|
|
zammbi
|
 |
«
Reply #28 - Posted
2009-12-02 06:50:50 » |
|
I say just ignore Java 1.4, or you could always make 2 versions but just submit the Java 1.5 version.
|
|
|
|
pjt33
|
 |
«
Reply #29 - Posted
2009-12-02 08:21:30 » |
|
I guess pack200 must also reduce some of the class overhead that comes with a standard java class.
IIRC at a minimum it exploits redundancy in the constant pool structure.
|
|
|
|
|