Java 4k Resources Thread

[Riven]

I might be wrong, and I'm probably looking at the wrong code, but I think I'm using session hijacking and a zeroday security hole.

I had a look for JavaScript but I couldn't find any, so I assumed you weren't actually hijacking the session. Hmm.

Packet sniffing doesn't show an actual session hijack, but the referrer should allow you to see who's requesting the image.

You think it's believable that I am guessing at how I did it? I thought it was obvious I was messing with you

[steveyO]

My Guess:  The link to Rivens avatar is a servlet or mapped to a servlet (not an image)..  The servlet picks up the user id (this is available from the http request in the referer, 'Live HTTP headers' plugin for Firefox is your friend here to see whats going on). Riven has a db/lookup table of usernames/ids.. And creates the animated gif in the Servlet and returns that.  Very clever!

[Riven]

My Guess:  The link to Rivens avatar is a servlet or mapped to a servlet (not an image)..  The servlet picks up the user id (this is available from the http request in the referer, 'Live HTTP headers' plugin for Firefox is your friend here to see whats going on)

Riven has a db/lookup table of usernames/ids.. And creates the animated gif in the Servlet and returns that.  Very clever!

I seriously cannot 'pick up a user id', that would be a major security hole in the browser, so I do not have IDs in my database - I only have your IP, the "Referer: --" header probably... and some other headers like the user-agent.

I also do not have an ip/username mapping in my database.

this is my stdout for a request:

1   2   3   4   5   6   7   8   9   10

requestReceived: /files/logo4.png addr: xxx.xxx.xxx.xxx username-score-1: steveyO (1.3125) username-score-2: zammbi  (0.3125) username-score-3: thijs   (0.2500) username-score-4: princec (0.2500) username-score-5: pjt33   (0.0625) username-score-6: Riven   (0.0625) whois took: 2ms >> reusing gif for: steveyO

Heuristics.

Games published by our own members! Go get 'em!

[moogie]

It is strange, i cannot seem to be able to replicate the results of Riven's fantastic tool.

I have narrowed it down the the proguard or the java source compilation steps as i am able to replicate the pack 200 step exactly and am able to match the kzip step.

It is not a major issue, i am just a little bemused! I will just use Riven's tool to produce the .pack version of my entry and then I will create my own gz version as I seem to be able to produce a more compressed .gz version.

My old 4kjo tool is now obsolete as proguard is able to optimise better than any combination of any other optimiser (as well as proguard in conjunction with any other optimiser) but I can leaver the recompression aspect of the tool and will write a tool that takes a .pack.gz as input and then recompress it to achieve better compression. (well most of the time any way!)

[pjt33]

I will create my own gz version as I seem to be able to produce a more compressed .gz version.

Which tool do you use for gzip?

[Riven]

It seems the pack200 is sensitive to the JAR it gets.

I'm creating the JAR with ZipFile and ZipEntry. I bet that qualifies as doMagicStuff()


Edit:
Hm, you say that your .pack is equal in size - nevermind then.

[Riven]

I did some more bruteforcing, and saved a handful of bytes.

old
A.normal.7z.pack.gz => 3201 bytes
A.progrd.7z.pack.gz => 3036 bytes
A.normal.kz.pack.gz => 3078 bytes
A.progrd.kz.pack.gz => 2934 bytes

new
A.normal.7z.pack.gz => 3201 bytes
A.progrd.7z.pack.gz => 3030 bytes
A.normal.kz.pack.gz => 3076 bytes
A.progrd.kz.pack.gz => 2927 bytes



What amazes me is that 7z is not deterministic.

[pjt33]

What amazes me is that 7z is not deterministic.

Nor is kzip. It has an option to randomise it, which it says is useful for running it lots of times and picking the smallest output, but even without that option the size of the file it produces can vary by a couple of bytes.

[Riven]

Nor is kzip. It has an option to randomise it, which it says is useful for running it lots of times and picking the smallest output, but even without that option the size of the file it produces can vary by a couple of bytes.

I already make use of it, since the last update actually

[moogie]

Which tool do you use for gzip?

multiple actually

I perform a similar brute force approach as Riven using kzip: multiple of 8 block size between 8 and 1024 and then for the "best" block size perform 30 random runs with the best block size. For each kzip run I then use DeflOpt which can sometimes shave an extra byte or two.

I then perform a brute force using BJWFlate given the same input to compress as kzip choosing split sizes as powers of 2 i.e. 4,8,16...8192. Similarly I use DeflOpt for each BJWFlate run.

The smallest resultant zip file out of all the runs is chosen to be passed to pjt33 zip to gzip converter. And thus the final gz is produced.

From previous years experience (and testing this year) some pack files are better compressed using kzip and some are better compressed using BJWflate.

Games published by our own members! Go get 'em!

[Riven]

multiple actually

I perform a similar brute force approach as Riven using kzip: multiple of 8 block size between 8 and 1024 and then for the "best" block size perform 30 random runs with the best block size. For each kzip run I then use DeflOpt which can sometimes shave an extra byte or two.

I then perform a brute force using BJWFlate given the same input to compress as kzip choosing split sizes as powers of 2 i.e. 4,8,16...8192. Similarly I use DeflOpt for each BJWFlate run.

The smallest resultant zip file out of all the runs is chosen to be passed to pjt33 zip to gzip converter. And thus the final gz is produced.

From previous years experience (and testing this year) some pack files are better compressed using kzip and some are better compressed using BJWflate.

What's the (highest) difference in byte count between the best kzip and the best [DeflOpt + BJWFlate] ?

[pjt33]

I then perform a brute force using BJWFlate given the same input to compress as kzip choosing split sizes as powers of 2 i.e. 4,8,16...8192. Similarly I use DeflOpt for each BJWFlate run.

According to the docs I've read DeflOpt is already included in BJWflate, so there's no extra benefit. Are you finding that to be incorrect?

I may have to install wine and play with BJWflate.

[pjt33]

BJWFlate.zip is no longer available on walbeehm.com. I managed to find it elsewhere, but I'd prefer to be cautious. Can anyone confirm the md5sum and cksum outputs?

md5sum: 95d7b11a281293045ae9c110881f8bf1  BJWFlate.exe
cksum: 492803991 39936 BJWFlate.exe

[Riven]

I was less cautious, so I'm already running it within Wine, with exactly the same md5sum.

N.B.: Also found it elsewhere, so that doesn't help you much

[Riven]

I did some more bruteforcing, and saved a handful of bytes.

old
A.normal.7z.pack.gz => 3201 bytes
A.progrd.7z.pack.gz => 3036 bytes
A.normal.kz.pack.gz => 3078 bytes
A.progrd.kz.pack.gz => 2934 bytes

new
A.normal.7z.pack.gz => 3201 bytes
A.progrd.7z.pack.gz => 3030 bytes
A.normal.kz.pack.gz => 3076 bytes
A.progrd.kz.pack.gz => 2927 bytes

Added support for BJWFlate:

A.normal.7z.pack.gz => 3201 bytes
A.progrd.7z.pack.gz => 3030 bytes
A.normal.kz.pack.gz => 3078 bytes
A.progrd.kz.pack.gz => 2930 bytes
A.normal.bj.pack.gz => 3107 bytes
A.progrd.bj.pack.gz => 2923 bytes

[Riven]

I added DeflOpt, and as pjt33 suspected, it didn't have any impact on filesize, so it's disabled for now.

[moogie]

I added DeflOpt, and as pjt33 suspected, it didn't have any impact on filesize, so it's disabled for now.

I guess i mistakenly "saw" a benefit with deflopt and bjwflate... but most likely i was mistaken

1   2   3   4   5   6   7   8   9   10   11   12

a.java => 11103 bytes a.class => 3940 bytes a.normal.jar => 2332 bytes a.progrd.jar => 1585 bytes a.normal.pack => 2558 bytes a.progrd.pack => 1636 bytes a.normal.7z.pack.gz => 1728 bytes a.progrd.7z.pack.gz => 1245 bytes a.normal.kz.pack.gz => 1665 bytes a.progrd.kz.pack.gz => 1226 bytes a.normal.bj.pack.gz => 1609 bytes a.progrd.bj.pack.gz => 1198 bytes

I just re-ran your updated online shrinking tool and it now compresses a further 28 bytes, very nice! now all that is left to be the ultimate tool for my purposes (he he) is for it to accept a JAR as input. I will be reading in a data file at run time so needs to be in the Jar.

excellent work!

I do have another idea on a completely different track of thought about how to shrink the size of the jar/pack.gz. It is a bit more complex and probably error prone but will be a nice challenge. Perhaps i should actually finish my java4k entry first

[Riven]

I just re-ran your updated online shrinking tool and it now compresses a further 28 bytes, very nice! now all that is left to be the ultimate tool for my purposes (he he) is for it to accept a JAR as input. I will be reading in a data file at run time so needs to be in the Jar.

Done!

[moogie]

fantastic! thanks

[Markus_Persson]

This is sooo obvious I can't believe it hasn't been brought up before:

Old:

1   2   3   4   5   6   7   8   9   10   11

public class M extends Applet implements Runnable {    private int[] M = new int[32767]; // Shared state    public void start() {       new Thread(this).start();    }    public void run() {              // Minecraft 4k game here        } }

2018 bytes.


But that adds M to the constant pool when run already is in there!
So what if we name the class and the member "run"?

New code:

1   2   3   4   5   6   7   8   9   10   11

public class run extends Applet implements Runnable {    private int[] run = new int[32767]; // Shared state    public void start() {       new Thread(this).start();    }    public void run() {              // Minecraft 4k game here        } }

2010 bytes.

[Abuse]

haha, n1

Though what does proguard do regarding the member?
Does it realize it's a shared name, or does it obfuscate it to a single letter?
Might have to run it with special parameters to prevent it obfuscating the member name?

[Markus_Persson]

Good question, I'm not using proguard yet in my build chain.
You can make proguard keep the game of the member, though, so it shouldn't be a problem.

[pjt33]

haha, n1

Though what does proguard do regarding the member?
Does it realize it's a shared name, or does it obfuscate it to a single letter?
Might have to run it with special parameters to prevent it obfuscating the member name?

It doesn't rename the class.

Ah. This wouldn't have worked before because of the jar file containing a longer name, but with pack200 and gzip the name of the class is stored only in the pack200's structures, so that's why this is suddenly better.

[Markus_Persson]

Are you sure? "jar.class" is only two letters longer than "M.jar", and the savings are more than 2 bytes.

[Abuse]

It doesn't rename the class.

I was talking about the member, i.e.

1	private int[] run = new int[32767]; // Shared state

I believe proguard will (unless you tell it not to) obfuscate this down to 'a', and in so doing reverse Markus's optimisation.

[Riven]

http://www.indiespot.net/app/java-four-kay

I added the option to keep all class members.

Maybe you should give it a try, my tiny examples turn out bigger, to I'm interested in the real deal.


normal Proguard config:

1   2   3   4   5   6   7

-libraryjars /root/jdk1.6.0_15/jre/lib/rt.jar -keep public class * -keep class * -allowaccessmodification -overloadaggressively -optimizationpasses 99 -defaultpackage ''

"keep members" Proguard config:

1   2   3   4   5   6   7   8   9   10   11   12   13   14

-libraryjars /root/jdk1.6.0_15/jre/lib/rt.jar -keep public class * -keep class * -allowaccessmodification -overloadaggressively -optimizationpasses 99 -defaultpackage '' -keepclassmembernames class * {   private *;   protected *;   public *;   * ; }

[pjt33]

I was talking about the member, i.e.

1	private int[] run = new int[32767]; // Shared state

I believe proguard will (unless you tell it not to) obfuscate this down to 'a', and in so doing reverse Markus's optimisation.

Oh. I use

1	-obfuscationdictionary proguard.dict

where proguard.dict contains

1   2   3   4   5   6   7

Code dispose isActive start sleep run a

a is there because it's the name of the class, but if you're changing that you'd want to drop it. That should be enough names for most Java4k projects, I think. If you need more just run javap over your class file to work out what methods you're using and update as appropriate.

[Eli Delventhal]

How are you guys using Riven's tool in terms of adding images and other resources into the jar?

[Riven]

How are you guys using Riven's tool in terms of adding images and other resources into the jar?

The diehards rolls their own.

[moogie]

How are you guys using Riven's tool in terms of adding images and other resources into the jar?

My graphics are animations of 8 frames each which consists of a global palette of 6 colours which of which sequential frames are simply XORed with the previous frame. Very cheap to decode and give sufficient  compression (when gzipped) due to the similarities between frames.

I have all my resources in a single separate file which is added to a jar with my class file and then i let Riven's tool at it