Indiespot compression not working

[StephR]

Starting with a log from 2011/11/28, I tried to find out which processes were undertaken by Riven's incomparable
Compile and shrink service, that most of us miss. What I found may be interesting.

First of all, I should indicate the configuration of my computer :
Windows 7 Home Premium - Service Pack 1
Java(TM) SE development Kit 6 Update 29 ( Windows x86 )
Proguard 4.6
Kzip compiled on 2007/04/14 ( latest release for what I know )

Kzip was the only archiver I used for this test.

And here is the log generated by Riven's tool on 2011/11/28 :
a.java => 24352 bytes
a.class => 9625 bytes
a.normal.java6.jar => 5194 bytes
a.normal.java6.pack => 7978 bytes
a.progrd.java6.jar => 5104 bytes
a.progrd.java6.pack => 7878 bytes
a.progrd.java6.kz.pack.gz => 4058 bytes
a.normal.java6.kz.pack.gz => 4078 bytes
a.progrd.java6.bj.pack.gz => 4086 bytes
a.progrd.java6.7z.pack.gz => 4280 bytes
a.normal.java6.7z.pack.gz => 4302 bytes

In the following lines, you will find the processes I set manually for the same source code as on 2011/11/28.

STEP 1 : JAVAC ( from java file )
Input file : a.java => 24352 bytes
Command line>javac -target 1.5 -g:none a.java
Output file : a.class => 9625 bytes ( C'n s: 9625 )

STEP 2 : JAR ( from step 1 class file )
Input file : a.class
Command line>jar cfM a.normal.java6.jar a.class
Output file : a.normal.java6.jar => 5199 bytes ( C'n s: 5194 )

STEP 3 : PACK200 ( from step 2 jar file )
Input file : a.normal.java6.jar
Command line>pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.normal.java6.pack a.normal.java6.jar  
Output file : a.normal.java6.pack => 7978 bytes ( C'n s: 7978 )

For this step, the command was supplied by Bonbon-Chan.

STEP 4 : PROGUARD ( from step 2 jar file )
Launch proguardgui.bat from Windows.

*Page Input/Output
Input file : a.normal.java6.jar
Output file : a.progrd.java6.jar

*Page Shrinking
Options
X Shrinking
Keep
X Applets
Also keep
none

*Page Obfuscation
Options
X Obfuscate
X Overload aggressively
Keep names
none

*Page Optimization
Options
X Optimize
X Allow access modification
9 Optimisation passes
Remove
all
Remove debugging
all

*Page Information
default settings

*Page Process
hit Process!

Output file : a.progrd.java6.jar => 5104 bytes ( C'n s: 5104 )

STEP 5 : PACK200 ( from step 4 proguarded jar file )
Input file : a.progrd.java6.jar
Command line>pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.progrd.java6.pack a.progrd.java6.jar  
Output file : a.progrd.java6.pack => 7878 bytes ( C'n s: 7878 )

STEP 6 : KZIP ( from step 5 proguarded and packed file )
Input file : a.progrd.java6.pack
Command line>kzip /b96 a.progrd.java6.kz.pack.gz a.progrd.java6.pack
Output file : a.progrd.java6.kz.pack.gz => 4177 bytes ( C'n s: 4058 )

For note, using the kzip default option /s0 ( Xtreme strategy ) gave worst results than /b96 with my file.

Kzip indicated that 4177 was the total size of the file, including headers. It also gave the mention "KSflating 4041".

STEP 7 : KZIP ( from step 3 packed file )
Input file : a.normal.java6.pack
Command line>kzip /b96 a.normal.java6.kz.pack.gz a.normal.java6.jar
Output file : a.normal.java6.kz.pack.gz => 4197 bytes ( C'n s: 4078 )

Kzip indicated that 4197 was the total size of the file, including headers. It also gave the mention "KSflating 4061".

*****************
Here is the comparison between Riven's log and my own results :

a.java => 24352 bytes Riven / 24352 bytes StephR
a.class => 9625 bytes Riven / 9625 bytes StephR ( step 1 )
a.normal.java6.jar => 5194 bytes Riven / 5199 bytes StephR ( step 2 )
a.normal.java6.pack => 7978 bytes Riven / 7978 bytes StephR ( step 3 )
a.progrd.java6.jar => 5104 bytes Riven / 5104 bytes StephR  ( step 4 )
a.progrd.java6.pack => 7878 bytes Riven / 7878 bytes StephR ( step 5 )
a.progrd.java6.kz.pack.gz => 4058 bytes Riven / 4177 bytes StephR  ( step 6 )
a.normal.java6.kz.pack.gz => 4078 bytes Riven / 4197 bytes StephR  ( step 7 )
a.progrd.java6.bj.pack.gz => 4086 bytes Riven / not evaluated
a.progrd.java6.7z.pack.gz => 4280 bytes Riven / not evaluated
a.normal.java6.7z.pack.gz => 4302 bytes Riven / not evaluated
*****************

CONCLUSION
It seems I misused Kzip, so I didn't get as good compression results as Riven's Compile and shrink service. Anyone could help to fix this ?

[SimonH]

I can't such good results either, maybe Riven has a 'special' kzip!

FWIW my build process goes;

1   2   3   4   5   6   7   8   9   10   11

javac -target 1.5 A.java jar cvfM A.normal.jar A.class java -jar jarg.jar A.normal.jar java -jar proguard.jar -injars A.normal_s.jar -outjars A.proguard.jar -libraryjars rt.jar -keep public class A -optimizationpasses 9 -overloadaggressively pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip A.proguard.pack A.proguard.jar kzip /b96 /y A.proguard.pack.gz A.proguard.pack

This takes my assassin game from a 12059 byte class file to a 4069 byte pack.gz file.

Strange phenomenon: If I give kzip the output filename 'A.pack.gz' instead of 'A.proguard.pack.gz' the file size grows by 1 byte!?

[Groboclown]

For my version of the kzip execution, I set the "random seed" command-line argument, and loop through executing it several times, and choose the smallest one.  Then, I loop through that bit, this time switching between different block sizes.  It takes a while (say, 2 minutes), but it's well worth the wait.

1	kzip /r /y /rn /s0 /b<block size> outfile.zip infiles

where <block size> is one of 0, 32, 64, 128, 256, 512, and 1024.

Games published by our own members! Check 'em out!

[bainit]

I additionally use JoGa after Proguard, it saves some bytes.

[SimonH]

For my version of the kzip execution, I set the "random seed" command-line argument, and loop through executing it several times, and choose the smallest one.  Then, I loop through that bit, this time switching between different block sizes.  It takes a while (say, 2 minutes), but it's well worth the wait.

1	kzip /r /y /rn /s0 /b<block size> outfile.zip infiles

where <block size> is one of 0, 32, 64, 128, 256, 512, and 1024.

I tried a brute force approach for kzip (every possible combination of switches) and found that in all the cases I tried /b96 gave the best results.

I additionally use JoGa after Proguard, it saves some bytes.

Good call! I'd forgotten JoGa - that shaved off another 43 bytes!

[Damocles]

But if you use KZip and just name the file .pack.gz its still a zip.

When I use unpack200, it just uses "copy-mode" and returns the same file.

Did you validate the result with unpack200 ?

[Groboclown]

But if you use KZip and just name the file .pack.gz its still a zip.

When I use unpack200, it just uses "copy-mode" and returns the same file.

Did you validate the result with unpack200 ?

You need to use a tool like this one to convert the zip file to a gz file:
http://www.java-gaming.org/topics/java-4k-resources-thread/21630/msg/177304/view.html#msg177304

[Damocles]

Ok, cool. Will try that out

[StephR]

Finally, and thanks to the comments of Bonbon-Chan, Groboclown, Damocles, SimonH and Bainit, I have reconstructed a complete compression chain similar in results to Riven's Compile and shrink service. Of course, unlike with this last, you have to do everything yourself !

To start, here are some informations about my operating system and the software I used :
Windows 7 Home Premium - Service Pack 1
Java(TM) SE development Kit 6 Update 29 ( Windows x86 )
Proguard 4.6
Kzip compiled on 2007/04/14 ( latest release for what I know )
Zip2Gzip ( pjt33's awesome converter )

And here is the comparison between Compile and shrink's log and my own results :

a.java => 24352 bytes C'n s / 24352 bytes StephR
a.class => 9625 bytes C'n s / 9625 bytes StephR
a.normal.java6.jar => 5194 bytes C'n s / 5199 bytes StephR
a.normal.java6.pack => 7978 bytes C'n s / 7978 bytes StephR
a.progrd.java6.jar => 5104 bytes C'n s / 5104 bytes StephR
a.progrd.java6.pack => 7878 bytes C'n s / 7878 bytes StephR
a.progrd.java6.kz.pack.gz => 4058 bytes C'n s / 4054 bytes StephR
a.normal.java6.kz.pack.gz => 4078 bytes C'n s / not evaluated
a.progrd.java6.bj.pack.gz => 4086 bytes C'n s / not evaluated
a.progrd.java6.7z.pack.gz => 4280 bytes C'n s / not evaluated
a.normal.java6.7z.pack.gz => 4302 bytes C'n s / not evaluated


In the following lines, you will find the processes for the complete compression chain :

STEP 1 : JAVAC ( from java file )
Input file : a.java => 24352 bytes
Command line>javac -target 1.5 -g:none a.java
Output file : a.class => 9625 bytes ( C'n s: 9625 )

STEP 2 : JAR
Input file : a.class
Command line>jar cfM a.normal.java6.jar a.class
Output file : a.normal.java6.jar => 5199 bytes ( C'n s: 5194 )

STEP 3: PROGUARD
Launch proguardgui.bat from Windows.

*Page Input/Output
Input file : a.normal.java6.jar
Output file : a.progrd.java6.jar

*Page Shrinking
Options
X Shrinking
Keep
X Applets
Also keep
none

*Page Obfuscation
Options
X Obfuscate
X Overload aggressively
Keep names
none

*Page Optimization
Options
X Optimize
X Allow access modification
9 Optimisation passes
Remove
all
Remove debugging
all

*Page Information
default settings

*Page Process
hit Process!

Output file : a.progrd.java6.jar => 5104 bytes ( C'n s: 5104 )

STEP 4 : PACK200 - REPACK
Input file : a.progrd.java6.jar
Command line>pack200 --repack a.progrd.java6.jar
Output file : a.progrd.java6.jar => 5116 bytes ( C'n s: not available )

STEP 5 : PACK200 - PACK
Input file : a.progrd.java6.jar
Command line>pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.progrd.java6.pack a.progrd.java6.jar  
Output file : a.progrd.java6.pack => 7878 bytes ( C'n s: 7878 )

For this step, the command was supplied by Bonbon-Chan.

STEP 6 : PACK200 - UNPACK
Input file : a.progrd.java6.pack
Command line>unpack200 a.progrd.java6.pack a.progrd.java6Test1.jar  
Output file : a.progrd.java6Test1.jar  => 5096 bytes ( C'n s: not available )

Here, you are supposed to test the resulting jar in a standard applet code. You can do it locally.

STEP 7 : KZIP
Input file : a.progrd.java6.pack
Command line>kzip /s0 /rn /b128 /y a.progrd.java6.kz.pack a.progrd.java6.pack
Output file : a.progrd.java6.kz.pack => 4172 bytes ( C'n s: not available )

As recommanded by Groboclown, here the kzip command should be repeated until you see what is the minimum compression value with the option /b128. Then, values other than 128 should be tested ( like /b64, /b96, /b104, /b112... ), and for each /bX value, the kzip command should be repeated again, until you find the minimum value with the optimal /bX value.

STEP 8 : ZIP2GZIP
Input file : a.progrd.java6.kz.pack
Command line>java Zip2Gzip a.progrd.java6.kz.pack a.progrd.java6.kz.pack.gz
Output file : a.progrd.java6.kz.pack.gz => 4054 bytes ( C'n s: 4058 )

Here, you will have to compile yourself the Zip2Gzip program before using it. Its source code can be found here, as indicated by Groboclown :
http://www.java-gaming.org/topics/java-4k-resources-thread/21630/msg/177304/view.html#msg177304.

STEP 9 : DEPLOYMENT
Now you can deploy your a.progrd.java6.kz.pack.gz file on your server, with the following applet code ( provided by appel ) put into a .htm page :

1   2   3

<applet code="a.class" archive="a.progrd.java6.kz.pack.gz" width="your width" height="your height"> <param name="java_arguments" value="-Djnlp.packEnabled=true" /> </applet>

FINAL NOTES
- the obsfuscator programs jarg ( to use before Proguard ) and JoGa ( to use after Proguard )  didn't help with my code. Maybe I misused them, or my code did not fit well for them ;
- about using pack200, I missed a few steps as I didn't sign my jar. But you will find some detailed informations about it here :
http://docs.oracle.com/javase/1.5.0/docs/guide/deployment/deployment-guide/pack200.html
http://docs.oracle.com/javase/6/docs/technotes/guides/jweb/tools/pack200.html
- it seems you can't test your pack.gz program without a server ;
- special thanks to pjt33 and his Zip2Gzip automagical converter.

[luisoft]

hello Riven, is the applet version ready? Maybe an offline version would be great so everyone could execute it in local machine and not depend on the service to be online.

Games published by our own members! Check 'em out!

[SimonH]

Thank you StephR! It's good to have this info available.
I don't think the repack stage is required though, unless you want to sign the file. It does add a few extra bytes after all!
So my build is now;

1   2   3   4   5   6   7   8   9   10   11   12   13

javac -target 1.5 A.java jar cvfM A.normal.jar A.class java -jar jarg.jar A.normal.jar java -jar proguard.jar -injars A.normal_s.jar -outjars A.proguard.jar -libraryjars rt.jar -keep public class A -optimizationpasses 9 -overloadaggressively pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip A.proguard.pack A.proguard.jar kzip /b96 /y A.proguard.kz.pack A.proguard.pack java Zip2Gzip A.proguard.kz.pack A.proguard.kz.pack.gz

A.class (12059) goes down to A.proguard.kz.pack.gz (3961)
I'd only tinker with /b96 if it was very close to 4096. I haven't yet found a case where it wasn't optimal for 4K.
I fiddled with JoGa but didn't get any reliable savings.

[ReBirth]

Ehhh it's my first time to play with things, SimonH's procedure seems easy but how to put that kz.pack.gz into applet? it's not executable is it?

[StephR]

+ SimonH, thank you for your interest and your own help. I have to do some more tests with jarg. I think I tried it along with JoGa, and maybe this one messed with jarg's result. Also, concerning the repack stage, by missing it, I remember my program didn't work in the end, but maybe the problem was not related to repacking. I wish I had more time to do more tests. I will certainly do so in the coming days, to see if I get better compressing results with jarg and without repacking.

+ ReBirth, here is the code to put the kz.pack.gz file into an applet :

1	<applet code="a.class" archive="a.progrd.kz.pack.gz" width="your width" height="your height"><param name="java_arguments" value="-Djnlp.packEnabled=true" /></applet>

It seems it doesn't work locally, unlike a no pack200 applet, but it would be normal. You have to try it on a server.

[SimonH]

+ SimonH, thank you for your interest and your own help. I have to do some more tests with jarg. I think I tried it along with JoGa, and maybe this one messed with jarg's result. Also, concerning the repack stage, by missing it, I remember my program didn't work in the end, but maybe the problem was not related to repacking. I wish I had more time to do more tests. I will certainly do so in the coming days, to see if I get better compressing results with jarg and without repacking.

+ ReBirth, here is the code to put the kz.pack.gz file into an applet :

1	<applet code="a.class" archive="a.progrd.kz.pack.gz" width="your width" height="your height"><param name="java_arguments" value="-Djnlp.packEnabled=true" /></applet>

It seems it doesn't work locally, unlike a no pack200 applet, but it would be normal. You have to try it on a server.

You can use pjt33's test server with the applet tag;

1   2   3   4   5   6   7

<applet     code=A     archive=A.proguard.kz     width=800     height=600>     <param name="java_arguments" value="-Djnlp.packEnabled=true"/> </applet>

Which works OK.
Generally though I just unpack200 the A.proguard.kz.pack.gz file and test the resulting jar - it's a bit less effort!

[Zarkonnen]

I'm having a lot of trouble getting pjt33's test server to work. I'm using this applet tag:

1   2   3   4

<applet code="a" archive="a.jar" width="800" height="600" border="1">   <param name="java_arguments" value="-Djnlp.packEnabled=true" />   Sorry, Java seems to be turned off or not supported in your browser. </applet>

When I try to load the page with the applet on it, I get the following output from the server:

1   2   3   4   5   6   7

Sat Dec 10 17:17:49 CET 2011 [/127.0.0.1] 200 (3966 bytes) for /a.jar.pack.gz Sat Dec 10 17:17:49 CET 2011 [/127.0.0.1] 200 (3966 bytes) for /a.jar.pack.gz Sat Dec 10 17:17:50 CET 2011 [/127.0.0.1] 200 (3966 bytes) for /a.jar Sat Dec 10 17:17:50 CET 2011 [/127.0.0.1] 200 (3966 bytes) for /a.jar.pack.gz Sat Dec 10 17:17:50 CET 2011 [/127.0.0.1] 200 (3966 bytes) for /a.jar Sat Dec 10 17:17:50 CET 2011 [/127.0.0.1] 404 (9 bytes) for /a.class Sat Dec 10 17:17:50 CET 2011 [/127.0.0.1] 404 (9 bytes) for /a.class

And a

1	java.lang.ClassNotFoundException: a

in the browser (Chrome). It looks like it successfully finds a.jar.pack.gz, but then wanders off and looks for the class a in /a.class instead.

Any suggestions on what might be wrong?

[SimonH]

Try the applet tag without the archive having either a .jar or .pack.gz extension (see my example above) as the server automatically adds it.

[Zarkonnen]

Try the applet tag without the archive having either a .jar or .pack.gz extension (see my example above) as the server automatically adds it.

Sadly not: the error is unchanged.

However, I just tried it with Cave4k's cave.pack.gz, and that does work, so clearly the error is with my jar file. The compression script I'm using (based off yours) goes like this:

1   2   3   4   5   6   7   8   9   10   11   12   13

cp ../src/a.java . javac -target 1.5 a.java jar cvfM a.normal.jar a.class java -jar ~/Downloads/proguard4.6/lib/proguard.jar -injars a.normal.jar -outjars a.proguard.jar -libraryjars /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar -keep public class a -optimizationpasses 9 -overloadaggressively pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.pack a.proguard.jar kzip -b96 -y a.pack.gz a.pack ls -la a.pack.gz

Maybe I made some mistake there? Thank you for all your help!

[Zarkonnen]

Ah, figured it out now! I wasn't actually turning the kzip into a gzip. My updated shown-to-work approach is:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15

cp ../src/a.java . javac -target 1.5 a.java jar cvfM a.normal.jar a.class java -jar ~/Downloads/proguard4.6/lib/proguard.jar -injars a.normal.jar -outjars a.proguard.jar -libraryjars /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar -keep public class a -optimizationpasses 9 -overloadaggressively pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.pack a.proguard.jar kzip -b96 -y a.kz.pack a.pack java Zip2Gzip a.kz.pack a.kz.pack.gz ls -la a.kz.pack.gz

[luisoft]

Guys, please help! For me the script is not working. Here is what I'm doing:

1   2   3   4   5   6   7   8   9   10   11   12   13

javac -target 1.5 a.java jar cvfM a.normal.jar a.class java -jar jarg.jar a.normal.jar java -jar proguard.jar -injars a.normal_s.jar -outjars a.proguard.jar -libraryjars /usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/rt.jar -keep public class a -optimizationpasses 9 -overloadaggressively pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.proguard.pack a.proguard.jar wine kzip /b96 /y a.proguard.kz.pack a.proguard.pack java Zip2Gzip a.proguard.kz.pack a.proguard.kz.pack.gz

And my applet html file:

1   2   3

<applet code="a.class" archive="a.proguard.kz.pack.gz" width="600" height="600"> <param name="java_arguments" value="-Djnlp.packEnabled=true" /> </applet>

When trying to run it on Firefox I got an exception that my a.class was not found... What I'm doing wrong?

[StephR]

luisoft, are you trying to run your applet locally ? Because if you do, it won't work because of pack200. Then, you should try to put your a.proguard.kz.pack.gz and your html files on a distant server.

[ra4king]

You don't use ".class" when specifying the code. Just use "a"

[Damocles]

Also try it out in chrome, in case firefox caches the previous jar.

[luisoft]

Tried all 3 suggestions above... neither worked!

I tried to put it in another machine running apache in the same network. Should it work?

I still get class not found error. I tried "a" without ".class" and it didn't work.

any other suggestion? how can I make sure the "a.class" file is in the packed file?

[Damocles]

use unpack200 to exptract the jar

then run it to validate

if it runs, it should be fine.
Else you can still reupload it on Java4k.com if something is not working

[StephR]

luisoft, instead of just this :

1	pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.proguard.pack a.proguard.jar

You should try this :

1   2   3   4   5

pack200 --repack a.proguard.jar pack200 --effort=9 --strip-debug --no-keep-file-order -O --no-gzip a.proguard.pack a.proguard.jar unpack200 a.proguard.pack a.proguard.test.jar

Then test locally the a.proguard.test.jar file, and if it works, continue on with kzip and your a.proguard.pack file.

[avarga]

Thanks for the compression tool Grobo. I eventually got it to work (there was one bug where the class was embedded in one extra directory but it's an easy fix) and now my game's well under 4k. 

[bysse]

I also made a python script for 'brute force' compression with KZip.
It tries different block sizes with random seed and takes the best one of them.
This usually gives me an additional 10-20 bytes.

EDIT: Ah this has already been mentioned, perfect!

[Morre]

Riven - I just wanted to reiterate how much we're missing your compression tool. Applet or server-side - either way would be greatly appreciated! Pretty please?

[tom]

With sugar on top!!

[Riven]

Riven - I just wanted to reiterate how much we're missing your compression tool. Applet or server-side - either way would be greatly appreciated! Pretty please?

I'm rather busy, and I doubt it will be useful to release in the last 6 days. Yes, I know, you'll be like 'but ofcouse it will be, especially in the last days' but I'm fairly sure everybody that wanted to write a java4k entry already rolled their own solution, based on the content in this thread.

The applet is 95% done, and has been like that for over a month now. Those last 5% will probably take half a day, and I don't really expect to be able to work on it in the next few days.