soszip2 - brute-force comression utility

[Sos]

Hi!
I believe this is my first post, and I hope it's a nice way to say hello.
Since some time, I'm developing an entry for this compo (but I'll make a thread when it's playable enough to show off) and I thought about making a custom zipping utility aimed at Java4k compo. Well, I guess I did it.

I made an LZMA (7zip algorithm) based compressor, which compresses a single file, storing only critical ZIP stuff inside. It's a dumb tool and uses brute-force trial and error way to get the best possible options. It does around million iterations, but for a 4k game it's usually enough to stop after a thousand. OH, And it saves the zip any time it finds it smaller than the last iteration, so you can break by pressing Escape at any time. Small help is included in a command line.

I hope it will come in handy, since making this just for myself would be a major waste of time.

Coolest screenshot ever:


Download (~30kb)

Cheers,
Sos

!!! UPDATE !!!
I added a small batch file ( sospack.bat ) that should strip some more redundant stuff using pack200 before the actual compression. You need to have JDK in your path for that. I'd love to hear some performance feedback on that one as well

PS. As a proof, I compressed the application with itself, so get yourself a 7-zip before downloading!
PS2. I don't compress app with itself anymore since I added a second file
PS3. Files produced by this are not intended for further stripping (mostly header stripping), but should work alright as-is. It's probably not as good as other tools around but it might prove better in some circumstances. Do try it.

[CaptainJester]

To be useful the resulting file must be able to be decompressed under standard ZIP compressions, since that is what the Java libraries use.

[SquashMonster]

I tried it out on Oh My Glob! but after an hour the resulting file didn't get any smaller than 6234 bytes, whereas it's compressing to 3662 bytes using KZip under Riven's tool.

Don't let this discourage you; I'm sure you'll get there if you keep working on it.

Games published by our own members! Check 'em out!

[Sos]

Well, I'm using LZMA SDK for compression, so I don't really have control over how's that part done. Yet, I build zip headers myself., and that saves a dozen of bytes compared to 7zip zip. I might tweak stuff but it might not get any better as well.

To be useful the resulting file must be able to be decompressed under standard ZIP compressions, since that is what the Java libraries use.

Well, yes it is! LZMA is a standard compresion method for ZIP, no worries!  Java runtime handles it rather easily too.

[Groboclown]

One thing to note about the LZMA compression is that it will only work for Jar files.  There's a method that some of us use for converting a zip compressed file into a gz file (for pack200 compression), and the gzip format (from what I've seen) doesn't support LZMA.

[Riven]

One thing to note about the LZMA compression is that it will only work for Jar files.  There's a method that some of us use for converting a zip compressed file into a gz file (for pack200 compression), and the gzip format (from what I've seen) doesn't support LZMA.

Zip = multiple entries of GZ

If a compression works in ZIP, it works in GZ.

In my compile-and-shrink service, I use 7z (which is LZMA) too for GZ files.

[Sos]

Frankly speaking, I haven't done any benchmarks and comparisons, but I'd love to see peoples results.
Also: Can this be used with your tool too, Riven? 

[Riven]

If your program has good results within about 5 seconds, then I might try it.

[Riven]

Trying to integrate it now...

[kappa]

oh wait, the jar format supports the lzma algorithm? since when?

Games published by our own members! Check 'em out!

[Riven]

No, 7z is using an algorithm like LZMA, yet ZIP compatible.
It works, as my shrinker output works

[Sos]

If your program has good results within about 5 seconds, then I might try it.

It usually gets best result within first 1000 tries, I can tweak it to do it within 5 seconds.

Also, ZIP specs:
http://www.pkware.com/documents/casestudies/APPNOTE.TXT

It states support for LZMA (method 14) in version 6.3. Yet, I guess many zippers would fail to decompress.
I tried creating a JAR and it runs just fine.

[Riven]

Hm... so Sos... seems you're really using LZMA, as opposed to 7z (which is something else than 7zip).

progrd.java6.s2.pack.gz => 3472 bytes  => test (this one doesn't work)
progrd.java6.bj.pack.gz => 3691 bytes  => test
progrd.java6.pk.pack.gz => 3698 bytes  => test
progrd.java6.7z.pack.gz => 3839 bytes  => test (existing 7z version that works)

The resulting files are smaller, but they are not zip compatible.

[Sos]

Wow! cool

I don't really know how your service works, but I think the issue is that gz is not exactly zip and output packed with my tool should be at the end of the chain, directly decompressed by JVM. I'll provide a timeout functionality today (think 2-3 hrs from now).

EDIT: Updated! Added timeout function. Specify number of seconds (integer) as a third parameter, no timeout if 0 or omitted. Original download link from the first post applies.

[Riven]

I don't really know how your service works, but I think the issue is that gz is not exactly zip and output packed with my tool should be at the end of the chain, directly decompressed by JVM. I'll provide a timeout functionality today (think 2-3 hrs from now).

Well... it is. If you don't believe me, read the ZIP spec.
You can simply strip data off of a zip and end up with a GZ.
It's how all other versions (kzip, bjwdeflate, 7z) are 'converted' to GZ in my service. So if yours doesn't work, it's either because your LZMA isn't zip compatible, or I have some silly bug in my code

I guess it's the former, as I can't extract your soszip2.zip with any other unzipper than 7z. Even WinRAR can't extract it.

[Sos]

WinRAR cannot extract LZMA zips at all, try to make one with 7-zip and see. I guess really not many archivers can do that (WinZip 12 claims to do so, I have to try it and see if it works).

 I can make it output raw LZMA with any additional info if you can integrate that. Just tell me what you need.

EDIT: WinZIP 15.0 does fine extracting my zips.

[Riven]

Anything that the JVM fails to load (non-GZ and non-ZIP) won't be of any use for compile-and-shrink.

... I thought about making a custom zipping utility aimed at Java4k compo...

I'm afraid your hard work isn't usable for Java4K



The Zip files your utility creates are (obviously) much larger than the stripped-to-gz zips that kzip/bjwdeflate/7z create, IIRC your generated zip was 4050 bytes, way bigger than the kzip GZ.

[kappa]

Sos just curious, can your tool be used generically just to produce small .lzma files (non 4k and zip stuff) ?

also what language is your tool written in? be useful to be able use it on non windows systems.

[Riven]

also what language is your tool written in? be useful to be able use it on non windows systems.

I run his exe on Linux through Wine.

[Sos]

I'll try to think something out. ZIP headers weigh 98 bytes + 2*filename length, so the overhead is actually relatively big.

It's written in C, actually file extension say cpp, since MSVC 6.0 name mangling is necromancy. And i might have thrown in some bools by accident
Oh! I use some WinAPI functions for getting file time/date and polling keyboard. But i guess it'd be minutes to port that. I can provide source, but it's really crappy, and I really don't have time to port it myself or to tidy this up but i might be of assistance if needed.

PS. Cool, it runs in wine So I guess I won't have to expose my crappy coding techniques

[kappa]

can't extract the zip file here on linux

1	skipping: soszip2.exe             need PK compat. v6.3 (can do v2.1)

edit: nvm, works with 7z not unzip

[kappa]

Sos just curious, can your tool be used generically just to produce small .lzma files (non 4k and zip stuff) ?

k, guess not, it only supports a single file and not a bunch of files or existing zip file. Also produces zip and not lzma.

[Sos]

I updated the tool with small batch file that uses pack200 to strip some data from the package. I hope this will give it some boost. The link remains in the first post.
Type sospack for usage info.

[pjt33]

You can simply strip data off of a zip and end up with a GZ.

You have to stick a gz header on too.

[Groboclown]

One thing to note about the LZMA compression is that it will only work for Jar files.  There's a method that some of us use for converting a zip compressed file into a gz file (for pack200 compression), and the gzip format (from what I've seen) doesn't support LZMA.

Zip = multiple entries of GZ

If a compression works in ZIP, it works in GZ.

In my compile-and-shrink service, I use 7z (which is LZMA) too for GZ files.

Perhaps I'm doing something wrong, or I'm misunderstanding something, but I just attempted to use the LZMA method in 7z to construct a jar (and, thus, also the gz file which has a stripped header):

> 7z a -bd -tzip -mx=9 -mm=LZMA -mtc=off -mcl=on -mcu=off test.jar D.class

I attempted to use the resulting jar file in Chrome, and the JVM fails with "Cannot load class D".  Using the Deflate method instead doesn't cause this error.

[Riven]

As said, the JVM doesn't support the LZMA algorithm, hence I removed it from compile-and-shrink.

[Groboclown]

As said, the JVM doesn't support the LZMA algorithm, hence I removed it from compile-and-shrink.

Ah, I overlooked that.  Thanks.