Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (476)
Games in Android Showcase (106)
games submitted by our members
Games in WIP (532)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  soszip2 - brute-force comression utility  (Read 5386 times)
0 Members and 1 Guest are viewing this topic.
Offline Sos

Senior Newbie


Projects: 1



« Posted 2010-12-09 01:05:10 »

Hi!
I believe this is my first post, and I hope it's a nice way to say hello.
Since some time, I'm developing an entry for this compo (but I'll make a thread when it's playable enough to show off) and I thought about making a custom zipping utility aimed at Java4k compo. Well, I guess I did it.

I made an LZMA (7zip algorithm) based compressor, which compresses a single file, storing only critical ZIP stuff inside. It's a dumb tool and uses brute-force trial and error way to get the best possible options. It does around million iterations, but for a 4k game it's usually enough to stop after a thousand. OH, And it saves the zip any time it finds it smaller than the last iteration, so you can break by pressing Escape at any time. Small help is included in a command line.

I hope it will come in handy, since making this just for myself would be a major waste of time.

Coolest screenshot ever:


Download (~30kb)

Cheers,
Sos

!!! UPDATE !!!
I added a small batch file ( sospack.bat ) that should strip some more redundant stuff using pack200 before the actual compression. You need to have JDK in your path for that. I'd love to hear some performance feedback on that one as well

PS. As a proof, I compressed the application with itself, so get yourself a 7-zip before downloading!
PS2. I don't compress app with itself anymore since I added a second file
PS3. Files produced by this are not intended for further stripping (mostly header stripping), but should work alright as-is. It's probably not as good as other tools around but it might prove better in some circumstances. Do try it.
Offline CaptainJester

JGO Knight


Medals: 12
Projects: 2
Exp: 14 years


Make it work; make it better.


« Reply #1 - Posted 2010-12-09 01:51:44 »

To be useful the resulting file must be able to be decompressed under standard ZIP compressions, since that is what the Java libraries use.

Offline SquashMonster

Senior Member


Medals: 1
Projects: 2



« Reply #2 - Posted 2010-12-09 02:42:10 »

I tried it out on Oh My Glob! but after an hour the resulting file didn't get any smaller than 6234 bytes, whereas it's compressing to 3662 bytes using KZip under Riven's tool.

Don't let this discourage you; I'm sure you'll get there if you keep working on it.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Sos

Senior Newbie


Projects: 1



« Reply #3 - Posted 2010-12-09 07:43:56 »

Well, I'm using LZMA SDK for compression, so I don't really have control over how's that part done. Yet, I build zip headers myself., and that saves a dozen of bytes compared to 7zip zip. I might tweak stuff but it might not get any better as well.

To be useful the resulting file must be able to be decompressed under standard ZIP compressions, since that is what the Java libraries use.

Well, yes it is! LZMA is a standard compresion method for ZIP, no worries!  Java runtime handles it rather easily too.
Offline Groboclown
« Reply #4 - Posted 2010-12-09 17:21:08 »

One thing to note about the LZMA compression is that it will only work for Jar files.  There's a method that some of us use for converting a zip compressed file into a gz file (for pack200 compression), and the gzip format (from what I've seen) doesn't support LZMA.

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #5 - Posted 2010-12-09 17:54:34 »

One thing to note about the LZMA compression is that it will only work for Jar files.  There's a method that some of us use for converting a zip compressed file into a gz file (for pack200 compression), and the gzip format (from what I've seen) doesn't support LZMA.

Zip = multiple entries of GZ

If a compression works in ZIP, it works in GZ.

In my compile-and-shrink service, I use 7z (which is LZMA) too for GZ files.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Sos

Senior Newbie


Projects: 1



« Reply #6 - Posted 2010-12-09 18:28:37 »

Frankly speaking, I haven't done any benchmarks and comparisons, but I'd love to see peoples results.
Also: Can this be used with your tool too, Riven? 
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #7 - Posted 2010-12-09 18:59:12 »

If your program has good results within about 5 seconds, then I might try it.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #8 - Posted 2010-12-09 20:24:47 »

Trying to integrate it now...

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline kappa
« League of Dukes »

JGO Kernel


Medals: 74
Projects: 15


★★★★★


« Reply #9 - Posted 2010-12-09 20:28:27 »

oh wait, the jar format supports the lzma algorithm? since when?
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #10 - Posted 2010-12-09 20:34:33 »

No, 7z is using an algorithm like LZMA, yet ZIP compatible.
It works, as my shrinker output works Smiley

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Sos

Senior Newbie


Projects: 1



« Reply #11 - Posted 2010-12-09 20:54:12 »

If your program has good results within about 5 seconds, then I might try it.

It usually gets best result within first 1000 tries, I can tweak it to do it within 5 seconds.

Also, ZIP specs:
http://www.pkware.com/documents/casestudies/APPNOTE.TXT

It states support for LZMA (method 14) in version 6.3. Yet, I guess many zippers would fail to decompress.
I tried creating a JAR and it runs just fine.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #12 - Posted 2010-12-09 20:58:51 »

Hm... so Sos... seems you're really using LZMA, as opposed to 7z (which is something else than 7zip).

progrd.java6.s2.pack.gz => 3472 bytes  => test (this one doesn't work)
progrd.java6.bj.pack.gz => 3691 bytes  => test
progrd.java6.pk.pack.gz => 3698 bytes  => test
progrd.java6.7z.pack.gz => 3839 bytes  => test (existing 7z version that works)


The resulting files are smaller, but they are not zip compatible. Undecided

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Sos

Senior Newbie


Projects: 1



« Reply #13 - Posted 2010-12-09 21:52:02 »

Wow! cool Wink

I don't really know how your service works, but I think the issue is that gz is not exactly zip and output packed with my tool should be at the end of the chain, directly decompressed by JVM. I'll provide a timeout functionality today (think 2-3 hrs from now).

EDIT: Updated! Added timeout function. Specify number of seconds (integer) as a third parameter, no timeout if 0 or omitted. Original download link from the first post applies.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #14 - Posted 2010-12-09 22:48:13 »

I don't really know how your service works, but I think the issue is that gz is not exactly zip and output packed with my tool should be at the end of the chain, directly decompressed by JVM. I'll provide a timeout functionality today (think 2-3 hrs from now).

Well... it is. If you don't believe me, read the ZIP spec.
You can simply strip data off of a zip and end up with a GZ.
It's how all other versions (kzip, bjwdeflate, 7z) are 'converted' to GZ in my service. So if yours doesn't work, it's either because your LZMA isn't zip compatible, or I have some silly bug in my code Smiley

I guess it's the former, as I can't extract your soszip2.zip with any other unzipper than 7z. Even WinRAR can't extract it.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Sos

Senior Newbie


Projects: 1



« Reply #15 - Posted 2010-12-10 00:31:34 »

WinRAR cannot extract LZMA zips at all, try to make one with 7-zip and see. I guess really not many archivers can do that (WinZip 12 claims to do so, I have to try it and see if it works).

 I can make it output raw LZMA with any additional info if you can integrate that. Just tell me what you need.

EDIT: WinZIP 15.0 does fine extracting my zips.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #16 - Posted 2010-12-10 00:37:18 »

Anything that the JVM fails to load (non-GZ and non-ZIP) won't be of any use for compile-and-shrink.

... I thought about making a custom zipping utility aimed at Java4k compo...

I'm afraid your hard work isn't usable for Java4K



The Zip files your utility creates are (obviously) much larger than the stripped-to-gz zips that kzip/bjwdeflate/7z create, IIRC your generated zip was 4050 bytes, way bigger than the kzip GZ.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline kappa
« League of Dukes »

JGO Kernel


Medals: 74
Projects: 15


★★★★★


« Reply #17 - Posted 2010-12-10 00:45:27 »

Sos just curious, can your tool be used generically just to produce small .lzma files (non 4k and zip stuff) ?

also what language is your tool written in? be useful to be able use it on non windows systems.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #18 - Posted 2010-12-10 00:54:43 »

also what language is your tool written in? be useful to be able use it on non windows systems.

I run his exe on Linux through Wine.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Sos

Senior Newbie


Projects: 1



« Reply #19 - Posted 2010-12-10 01:01:09 »

Sad
I'll try to think something out. ZIP headers weigh 98 bytes + 2*filename length, so the overhead is actually relatively big.

It's written in C, actually file extension say cpp, since MSVC 6.0 name mangling is necromancy. And i might have thrown in some bools by accident Tongue
Oh! I use some WinAPI functions for getting file time/date and polling keyboard. But i guess it'd be minutes to port that. I can provide source, but it's really crappy, and I really don't have time to port it myself or to tidy this up but i might be of assistance if needed.

PS. Cool, it runs in wine Wink So I guess I won't have to expose my crappy coding techniques Tongue

Offline kappa
« League of Dukes »

JGO Kernel


Medals: 74
Projects: 15


★★★★★


« Reply #20 - Posted 2010-12-10 01:10:09 »

can't extract the zip file here on linux Smiley
1  
skipping: soszip2.exe             need PK compat. v6.3 (can do v2.1)


edit: nvm, works with 7z not unzip
Offline kappa
« League of Dukes »

JGO Kernel


Medals: 74
Projects: 15


★★★★★


« Reply #21 - Posted 2010-12-10 01:18:26 »

Sos just curious, can your tool be used generically just to produce small .lzma files (non 4k and zip stuff) ?

k, guess not, it only supports a single file and not a bunch of files or existing zip file. Also produces zip and not lzma.
Offline Sos

Senior Newbie


Projects: 1



« Reply #22 - Posted 2010-12-10 02:02:07 »

I updated the tool with small batch file that uses pack200 to strip some data from the package. I hope this will give it some boost. The link remains in the first post.
Type sospack for usage info.
Offline pjt33
« Reply #23 - Posted 2010-12-10 11:53:13 »

You can simply strip data off of a zip and end up with a GZ.
You have to stick a gz header on too.
Offline Groboclown
« Reply #24 - Posted 2010-12-17 16:31:46 »

One thing to note about the LZMA compression is that it will only work for Jar files.  There's a method that some of us use for converting a zip compressed file into a gz file (for pack200 compression), and the gzip format (from what I've seen) doesn't support LZMA.

Zip = multiple entries of GZ

If a compression works in ZIP, it works in GZ.

In my compile-and-shrink service, I use 7z (which is LZMA) too for GZ files.

Perhaps I'm doing something wrong, or I'm misunderstanding something, but I just attempted to use the LZMA method in 7z to construct a jar (and, thus, also the gz file which has a stripped header):

> 7z a -bd -tzip -mx=9 -mm=LZMA -mtc=off -mcl=on -mcu=off test.jar D.class

I attempted to use the resulting jar file in Chrome, and the JVM fails with "Cannot load class D".  Using the Deflate method instead doesn't cause this error.

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 742
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #25 - Posted 2010-12-17 16:35:07 »

As said, the JVM doesn't support the LZMA algorithm, hence I removed it from compile-and-shrink.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Groboclown
« Reply #26 - Posted 2010-12-17 16:39:26 »

As said, the JVM doesn't support the LZMA algorithm, hence I removed it from compile-and-shrink.
Ah, I overlooked that.  Thanks.

Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

pw (14 views)
2014-07-24 01:59:36

Riven (14 views)
2014-07-23 21:16:32

Riven (13 views)
2014-07-23 21:07:15

Riven (15 views)
2014-07-23 20:56:16

ctomni231 (43 views)
2014-07-18 06:55:21

Zero Volt (40 views)
2014-07-17 23:47:54

danieldean (32 views)
2014-07-17 23:41:23

MustardPeter (36 views)
2014-07-16 23:30:00

Cero (50 views)
2014-07-16 00:42:17

Riven (50 views)
2014-07-14 18:02:53
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!