Adding resources to the class path at run time...

[moogie]

I have made a little utility which wraps a given JAR file in a Bzip2 compression wrapper giving smaller JAR file sizes while keeping the same functionality.

It can be downloaded here: http://www.geocities.com/budgetanime/bJAR.html

It currently performs as excpected with Executable JARs. It does not work with normal Library JARs. In order to make library JARs work I need to be able to add resources to the class path from a static method. My current code which adds resources to the class path only works from the main thread. i.e. from the public static void main (String args[]) method or any called methods.

The code which adds the compressed JAR to the class path:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80   81   82   83   84   85   86   87   88   89   90   91   92   93   94   95

import java.io.*; import java.lang.reflect.*; import java.net.*;     public class bJARLoader {     static private bJARLoader instance;         static     {         instance=new bJARLoader();         //        URL dataURL=instance.getClass().getResource("data.bz2"); //        URL.setURLStreamHandlerFactory(new ExtendedStreamHandlerFactory(dataURL)); //         //        try //        { //            addURL(new URL("bjar:data.jar")); //        } //        catch (IOException e) //        { //            e.printStackTrace(); //            System.exit(1); //        }             }         static final int BUFFER = 2048;     public static void main(String[] args) throws Exception     {         URL infoURL=instance.getClass().getResource("bJAR.info");         URL dataURL=instance.getClass().getResource("data.bz2");                 URL.setURLStreamHandlerFactory(new ExtendedStreamHandlerFactory(dataURL));         addURL(new URL("bjar:data.jar"));         BufferedReader br = new BufferedReader(new InputStreamReader(infoURL.openStream()));         String startClassName = br.readLine();         if (startClassName!=null)         {             startClassName=startClassName.substring("Main-Class: ".length());             runActualJar(startClassName,args);         }     }           // The methods addFile and associated final Class[] parameters were    // gratefully copied from      // anthony_miguel @ http://forum.java.sun.com/thread.jsp?forum=32&thread=300557&tstart=0&trange=15      private static final Class[] parameters = new Class[]{URL.class};           public static void addFile(String s) throws IOException {          File f = new File(s);          addFile(f);       }//end method           public static void addFile(File f) throws IOException {          addURL(f.toURL());       }//end method           public static void addURL(URL u) throws IOException {          URLClassLoader sysloader = (URLClassLoader) ClassLoader.getSystemClassLoader();          Class sysclass = URLClassLoader.class;          try {             Method method = sysclass.getDeclaredMethod("addURL",parameters);             method.setAccessible(true);             method.invoke(sysloader,new Object[]{ u });          } catch (Throwable t) {             t.printStackTrace();             throw new IOException("Error, could not add URL to system classloader");          }//end try catch      }//end method     // runs the main method of the compressed JAR    private static void runActualJar(String startClassName, String[] variables) throws Exception     {         Class startClass = Class.forName(startClassName);         Class[] sig = new Class[1];         sig[0] = String[].class;         Object[] param = new Object[1];         param[0] = variables;         Method bb = startClass.getMethod("main", sig);         Object cc = bb.invoke(null, param);     } }

The actual adding occurs in the AddURL() method.

When i de-comment the block of code in the static block at the top of the listing hoping to add the JAR contents to the class path before instaniation, there is an exception indicating that the addURL() method for the system class loader  does not exisit. This method does seem to exist when called from the main method.

My question is how can i add resources to the main system class loader from a static block of code....

[Jeff]

Nope I dont think so.

My off the cuff reaction is that would open up nasty security holes.

If I can udnerstand what you are trying to do maybe Ican offer a better approach.  At the moment Im not entirely clear.

[moogie]

Hmm... Thats what i was afraid of...

What i am trying to do is to make a new functionally identical JAR from a given input JAR but instead of using DEFLATE to compress the JAR it uses the Bzip2 Burrows Wheeler Block Matching compression.

It currently works well for an Executable JAR because of the fact there is an instaniation of a class and thus the system class loader has a "addURL" method which will add classes ,or a JAR in this case, to the classpath at runtime. This allows me to decompress the stored original JAR and add it to the system class loader. I then ultimately invoke the main method of the sorted original JAR.

i.e. this is how it currently works:

Making the bzip2 output jar:

1. convert the input jar to a temporary jar which is using the "STORE" compression technique.
2. compress the temp jar using bzip2 streams. (step 1 and 2 are performed in memory using streams)
3. copy the input jar's manifest and place it into the root directory of the output jar
4. copy the classes needed for decompressing the bzip stream into the output jar
5. copy the bzip2 compressed input jar into the output jar

running the bzip2 output jar:

1. read the stored manifest file and add the "class-path" resources to the system class loader if there is any external JARs needed
2. read the stored manifest file and find the main class.
3. decompress and add the bzip2 compressed input jar to the system class loader.
4. invoke the main method of the main class.

The problem exists that it does not work with "library jars" as they have static methods which are called. and i also need to have some sort of method protoypes in the output jar so that programs that use the input jar will be able to use the output jar as they need the same method signature.

Coupled with the fact that it seems that the system class loader does not have the "addURL" method when the method is invoked from static methods

Games published by our own members! Check 'em out!

[blahblahblahh]

The problem exists that it does not work with "library jars" as they have static methods which are called. and i also need to have some sort of method protoypes in the output jar so that programs that use the input jar will be able to use the output jar as they need the same method signature.

Coupled with the fact that it seems that the system class loader does not have the "addURL" method when the method is invoked from static methods

The problem is that this is java, not C++, and java is specifically and deliberately designed not to let you do things you might want to do, but only to let you do things that the original designers decided you ought to do. In C++, you can transparently override or hook into practically anything - as often said, C++ is not an actual language, it is a meta-language that you can turn into *any* language, because anything and everything can be customized.

Your customization is desirable, and I personally would like it , but like many many other great things people have come up with, it can't be used until Sun chooses to accept it and does it themselves, unless you really hack java to pieces, at which point most people will lose interest. This is one of the downsides of java, and you just have to live with it .

[moogie]

In a flash of insight I have managed to get around the "system class loader does not have the "addURL" method when the method is invoked from static methods" problem.

It is a little contrived but it does seem to work. I am now launching  a thread from a static block of code which adds the jar to the class path. The main thread pauses until this jar-adding thread is complete.

now all i need to do is figure out how to make protoype classes from the classes in the input JAR. Is the class byte code definiton setup so that i can simply copy only the "header" infomation with out the logic?

[oNyx]

Zip isnt all that great, right.

For what do you need better compression? Distribution of course. And there are already nice solutions for those common distribution ways. LZMA [+pack200] for installers and pack200/gzip for webstart (1.5+).

For example the lwjgl.jar (331kb) gets down to 61kb with p200/gzip compression.

Some days ago I rewrote my p200/gzip php/htaccess stuff. I'm pretty satisfied with the new version (there were several issues with the previous version)... I think I'll write an article soon.

You can try a silly demonstration here:
http://kaioa.com/src/tmp/BounceDemo2.jnlp
[It merely tells you in which form the archive traveled to you. Eg "I was a .jar" on 1.4.x jws clients and "I was a .jar.pack.gz" on 1.5+]

[moogie]

Sure, i know i am probably re-inventing the wheel, but it is a challenge and i have not had much to do with class loaders and the like. It is more of an educational exercise for me.

there are some benefits of what i am doing:

1. it is compatible with java 1.2+ (i beleive, i know it is compatible with 1.4)
2. it allows for the option of even better compression algorithms later on. (i.e. it may be possible to use pack200 (LZMA) so that previous versions of java can get the same compression benefit as 1.5)
3. gives me a better understanding of how java works behind the scenes.

[Jeff]

Perfectly reasonable goal in Java.

I suspect you simply attacked it the wrong way.

URLCLassLoader doesnt care what format the data is in, as long as it can be
expressed as a URL.  There are URL handlers in the stndard library for all the usual kinds of
URLs, such as the jar.

The easiest way to extend class laoding types (or data formats in general) is simply to define a new URL type and
URL handler. 

[moogie]

The easiest way to extend class laoding types (or data formats in general) is simply to define a new URL type and
URL handler. 

Yep, that is exactly what i have done. i have created a "Bzip2StreamHandler" which implements URLStreamHandler.

I have also set the URLStreamHandlerFactory in the URL to a Stream Handler factory which i have created.


On a side note i have found a java implementation of LZMA compression... i think i will change the program so it has a more plug-in architecture for compressors.

[blahblahblahh]

URLCLassLoader doesnt care what format the data is in, as long as it can be
expressed as a URL.  There are URL handlers in the stndard library for all the usual kinds of
URLs, such as the jar.

And you don't get any problems with the instanceof bug^H^H^Hfeature?

I'm betting you do, in fact, and that it buggers up your code. Especially libraries! If not, well, fantastic, and doh! to me...

(I'd assumed this approach had been rejected already for this reason)

Games published by our own members! Check 'em out!

[Jeff]

URLCLassLoader doesnt care what format the data is in, as long as it can be
expressed as a URL.  There are URL handlers in the stndard library for all the usual kinds of
URLs, such as the jar.

And you don't get any problems with the instanceof bug^H^H^Hfeature?

I dont knwo what bug youa re erring to but the one time I did this. no i had no trouble with it atall.

(It was for the worlds smallest EJB container, a reserach project, and the loader understood an extended form of the jar! type URL to allow it to load things from sub-jars within a jar file.)

[moogie]

I believe i have almost at a stage where i can keep the functionality of a "library" JAR... however i have hit a stumbling block...

I seem not to be able to access some private fields from a subclass whoes parent it abstract...

here is some code to demonstrate this: (based on yawmark's code)

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30

import java.lang.reflect.*; import java.net.URLClassLoader; import java.io.*; import java.net.*;   class PrivateReflection {       public static void main(String[] args) throws Exception {         Private p = new Private();         File f=new File("test.jar");         URL[] urls=new URL[1];         urls[0]=f.toURL();         URLClassLoader loader=new URLClassLoader(urls);         Class c=loader.getClass(); //        Class c=p.getClass();               Field field = c.getField("packages");         field.setAccessible(true);         System.out.println(field.get(p));     } }   class Private {     private String packages = "Can't get to me!";           private void privateMethod() {         System.out.println("The password is swordfish");     } }

There is no need for a test.jar file to be created.
I know for a fact that the ClassLoader abstract class defines a field called "packages" which is an HashMap.

How do i access this private field?

[moogie]

well, i managed to remove classes and jars from the system class loader and replace them with the decompressed versions, however it seems that the cost of having to include the prototype classes with the output JAR negates the savings from using better compression.

Unless there is a way to dynamically create the class prototypes when the jar is first loaded into the system class loader I believe that it is not worth persuing re-compressing "library JARs". However it the re-compression of executable JARs is definitely feasiable, especially with LZMA compression.