theagentd
JGO Wizard  Posts: 1392 Medals: 88
|
 |
«
on:
2011-11-12 03:46:42 » |
|
Okay, I've got to the point where I want to implement a scripting language. I want a very powerful scripting language, but it obviously has to be fast, so I thought to myself: "What's wrong with Java?". I mean, I can compile the scripts at runtime to bytecode, load it and run it at the same speed as normal code. It should beat any scripting language implementation when it comes to performance, plus be easier to implement. I realize that I will need some good security for this. I need to disallow networking, reflection, file access, e.t.c, or people could create viruses in my game scripts!  I'm also thinking of restricting class access to a white list. I've heard that these 2 things can be implemented with a SecurityManager and a custom ClassLoader, respectively. Now, I have no experience what so ever with SecurityManagers and ClassLoaders and such stuff, so I don't really know how to use them. I've managed to compile some test Java code with JavaCompiler, but I don't really understand how to load the class after that. I've tried Google, but I couldn't find anything like this... |
There is no god.
|
|
|
divxdede
Jr. Member Posts: 58
|
|
«
Reply #1 on:
2011-11-12 04:06:30 » |
|
You should go with "scripting" feature of Java with a langage like Groovy that can be easily integrated inside a java program. you can read this : http://groovy.codehaus.org/Embedding+Groovyand do code like it in your java program 1
2
3
4
5
6
7
8
| ClassLoader parent = getClass().getClassLoader();
GroovyClassLoader loader = new GroovyClassLoader(parent);
Class groovyClass = loader.parseClass(new File("src/test/groovy/script/HelloWorld.groovy"));
GroovyObject groovyObject = (GroovyObject) groovyClass.newInstance();
Object[] args = {};
groovyObject.invokeMethod("run", args); |
|
|
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #2 on:
2011-11-12 04:35:30 » |
|
Interesting! It even seems to be able to do object orientation too, which I will need! I'll look into it, but it seems a little complicated...
|
There is no god.
|
|
|
Games published by our own members! Go get 'em! 
|
|
nsigma
Sr. Member Posts: 342 Medals: 18
|
|
«
Reply #3 on:
2011-11-12 05:38:00 » |
|
You could have a look at Janino http://docs.codehaus.org/display/JANINO/HomeAs well as being an embedded compiler, which has the advantage that it will work with the JRE, it can now also wrap javac. It manages the ClassLoader stuff for you, and it is possible to set a ProtectionDomain on the loaded classes. It also provides the means to compile individual methods or expressions. I use Janino as the basis for the live compilation in Praxis and it works really well. I haven't yet looked at the security manager aspect of it - would be nice but not essential for my use case - if the user wants to destroy their home directory, I'm not going to stop them!  |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #4 on:
2011-11-12 08:19:30 » |
|
You could have a look at Janino http://docs.codehaus.org/display/JANINO/HomeAs well as being an embedded compiler, which has the advantage that it will work with the JRE, it can now also wrap javac. It manages the ClassLoader stuff for you, and it is possible to set a ProtectionDomain on the loaded classes. It also provides the means to compile individual methods or expressions. I use Janino as the basis for the live compilation in Praxis and it works really well. I haven't yet looked at the security manager aspect of it - would be nice but not essential for my use case - if the user wants to destroy their home directory, I'm not going to stop them! This looks just perfect. Nyehehehe... (<--- my experimenting laugh) Groovy looks interesting too, but I'll start with Janino for now. EDIT: This code freaks me out. 1
2
3
4
5
6
7
8
9
10
11
12
13
14
| package dam.test;
import org.codehaus.janino.ExpressionEvaluator;
public class JaninoTest {
public static void main(String[] args) throws Exception{
ExpressionEvaluator exp = new ExpressionEvaluator("10 + 5*3 + dam.test.JaninoTest.getValue(\"stat\")", int.class, new String[0], new Class[0]);
System.out.println(exp.evaluate(new Object[0]));
}
public static int getValue(String name){
return 5;
}
} |
|
There is no god.
|
|
|
nsigma
Sr. Member Posts: 342 Medals: 18
|
|
«
Reply #5 on:
2011-11-12 09:46:24 » |
|
EDIT: This code freaks me out.
Why? Is that not exactly what you'd expect? |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #6 on:
2011-11-12 10:30:28 » |
|
Well, yeah, but the power is overwhelming. xD Hahaha...
Okay, so I've got my little compile test running, and I've also made a custom SecurityManager that denies everything (I'll make it toggle-able eventually). Is there any way to restrict access to a whitelist of classes? I'd like to restrict access from everything that isn't needed. This is to serve as an extra security layer (paranoia xD), but also as a hint to scripters to keep the scripts simple. Basically to send the message that if you can't do it with the available classes, you're either not supposed to be doing that, or you're Doing It Wrong. I think I heard someone mentioning that this could be done with a custom ClassLoader, but I don't see how to do that at all. How do I even create a ClassLoader? T___T
|
There is no god.
|
|
|
nsigma
Sr. Member Posts: 342 Medals: 18
|
|
«
Reply #7 on:
2011-11-12 11:01:48 » |
|
Well, yeah, but the power is overwhelming. xD Hahaha...
Okay, so I've got my little compile test running, and I've also made a custom SecurityManager that denies everything (I'll make it toggle-able eventually). Is there any way to restrict access to a whitelist of classes? I'd like to restrict access from everything that isn't needed. This is to serve as an extra security layer (paranoia xD), but also as a hint to scripters to keep the scripts simple. Basically to send the message that if you can't do it with the available classes, you're either not supposed to be doing that, or you're Doing It Wrong. I think I heard someone mentioning that this could be done with a custom ClassLoader, but I don't see how to do that at all. How do I even create a ClassLoader? T___T
hmmm ..... power! ......  It might be worth looking at this bug report http://jira.codehaus.org/browse/JANINO-66 The Sandbox attachment might be useful in terms of setting up an unprivileged context. Along with that, the mentioned auxillary classes are set in SimpleCompiler.setParentClassLoader(..) which is a superclass of ExpressionEvaluator (and the other evaluators). You should be able to restrict access to just JVM classes using the BOOT_CLASS_LOADER field in SimpleCompiler. It may even be possible to pass a ClassLoader that doesn't load anything, and pass all the classes you want people to be able to load in the auxillary classes array. As I said before, I haven't tried any of these things out myself yet. Good luck!  |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #8 on:
2011-11-12 11:51:46 » |
|
I'm using JavaSourceClassLoader to load .java files, and it actually had a ClassLoader constructor argument. I did this: 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| private class RestrictedClassLoader extends ClassLoader {
public RestrictedClassLoader(ClassLoader parent){
super(parent);
}
@Override
public Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
System.out.println("Loading class " + name);
if(
!name.startsWith("java.lang.") && !name.equals("java.io.Serializable") && !name.startsWith("compiling.") && !name.startsWith("dam.test.") ){
throw new SecurityException("Bad class!");
}
return super.loadClass(name, resolve);
}
} |
It seems to be working. I also added the SecurityManager thingy, and it all seems to work fine. I'll have to make a "Hack my script engine" contest before I release the game if I ever get that far. xD Thanks for your help Nsigma, and thanks to Divxdede too! |
There is no god.
|
|
|
Riven
« League of Dukes » JGO Kernel  Posts: 5867 Medals: 255
Hand over your head.
|
|
«
Reply #9 on:
2011-11-12 12:23:59 » |
|
You forgot to exclude reflection  |
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings |
|
|
Games published by our own members! Go get 'em! 
|
|
avm1979
Full Member Posts: 157 Medals: 10
|
|
«
Reply #10 on:
2011-11-12 12:25:49 » |
|
Sorry to hijack the thread a bit... I've been using Janino for a while, and ran into an odd bug - it seems to not be able to handle break/continue inside a loop. It will compile that code fine, but then dies on the next compilation unit with this stack trace: 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| org.codehaus.janino.JaninoRuntimeException: Cannot "set()" Offset more than once
at org.codehaus.janino.CodeContext$Offset.set(CodeContext.java:1178)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1009)
at org.codehaus.janino.UnitCompiler.access$1000(UnitCompiler.java:104)
at org.codehaus.janino.UnitCompiler$5.visitForStatement(UnitCompiler.java:870)
at org.codehaus.janino.Java$ForStatement.accept(Java.java:1537)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:888)
at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:914)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1999)
at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:789)
at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:770)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:464)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:357)
at org.codehaus.janino.UnitCompiler$3.visitPackageMemberClassDeclaration(UnitCompiler.java:312)
at org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:770)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:319)
at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:288)
at org.codehaus.janino.JavaSourceClassLoader.generateBytecodes(JavaSourceClassLoader.java:203)
at org.codehaus.janino.JavaSourceClassLoader.findClass(JavaSourceClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at com.fs.starfarer.loading.scripts.ScriptStore$1.run(ScriptStore.java:85) |
Has anyone had much experience with it and managed to work around the problem (or not run into it at all)? |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #11 on:
2011-11-12 13:53:18 » |
|
You forgot to exclude reflection Excluded with the SecurityManager, but I'll be using a whitelist later. Sorry to hijack the thread a bit... I've been using Janino for a while, and ran into an odd bug - it seems to not be able to handle break/continue inside a loop. It will compile that code fine, but then dies on the next compilation unit with this stack trace: 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| org.codehaus.janino.JaninoRuntimeException: Cannot "set()" Offset more than once
at org.codehaus.janino.CodeContext$Offset.set(CodeContext.java:1178)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1009)
at org.codehaus.janino.UnitCompiler.access$1000(UnitCompiler.java:104)
at org.codehaus.janino.UnitCompiler$5.visitForStatement(UnitCompiler.java:870)
at org.codehaus.janino.Java$ForStatement.accept(Java.java:1537)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:888)
at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:914)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1999)
at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:789)
at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:770)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:464)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:357)
at org.codehaus.janino.UnitCompiler$3.visitPackageMemberClassDeclaration(UnitCompiler.java:312)
at org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:770)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:319)
at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:288)
at org.codehaus.janino.JavaSourceClassLoader.generateBytecodes(JavaSourceClassLoader.java:203)
at org.codehaus.janino.JavaSourceClassLoader.findClass(JavaSourceClassLoader.java:157)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at com.fs.starfarer.loading.scripts.ScriptStore$1.run(ScriptStore.java:85) |
Has anyone had much experience with it and managed to work around the problem (or not run into it at all)? Uwaaaa... Sounds horrible! Keep me updated!!! |
There is no god.
|
|
|
Riven
« League of Dukes » JGO Kernel Posts: 5867 Medals: 255
Hand over your head.
|
|
«
Reply #12 on:
2011-11-12 17:37:56 » |
|
You forgot to exclude reflection Excluded with the SecurityManager, but I'll be using a whitelist later. You can use reflection with any security manager. The security manager is only notified when you do something 'forbidden' like accessing a private field. |
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings |
|
|
Mads
JGO Ninja Posts: 674 Medals: 16
Directly directional
|
|
«
Reply #13 on:
2011-11-13 01:04:30 » |
|
What about javascript? You can implement it almost exactly like java, and it will take parameters. Does not need compilimg.
|
|
|
|
Nate
JGO Neuromancer Posts: 1062 Medals: 30
mooooo
|
|
«
Reply #14 on:
2011-11-13 03:28:47 » |
|
+1 Janino. -1 Groovy (pile of junk).
|
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #15 on:
2011-11-13 04:00:47 » |
|
You forgot to exclude reflection Excluded with the SecurityManager, but I'll be using a whitelist later. You can use reflection with any security manager. The security manager is only notified when you do something 'forbidden' like accessing a private field. I'll use a whitelist in the end anyway, so it should be okay. That code was just for my test. What about javascript? You can implement it almost exactly like java, and it will take parameters. Does not need compilimg.
It needs to be fast, so not having compilation needing compilation is a bad thing in this case. @Avm1979 I managed to reproduce the error. Goddamn it! Is there a bug report for this problem already, or should we hurry up and make one? EDIT: Yay! http://jira.codehaus.org/browse/JANINO-147 |
There is no god.
|
|
|
nsigma
Sr. Member Posts: 342 Medals: 18
|
|
«
Reply #16 on:
2011-11-13 07:02:08 » |
|
You forgot to exclude reflection Excluded with the SecurityManager, but I'll be using a whitelist later. You can use reflection with any security manager. The security manager is only notified when you do something 'forbidden' like accessing a private field. @Riven. Can you clarify what the issue with reflection would be? How could you access anything that you couldn't with normal code anyway, as the SecurityManager will (should) stop you accessing other classloaders / private fields? This is a sincere question, btw - it's not an area of Java I feel I know a lot about. Dang, was just going to reply and point you to that! Seems fixed in 2.6.2, but that's not released yet. Incidentally, could you post an example that will reproduce this. Have tried in Praxis but can't seem to force this error - incidentally, that's with the last release of the 2.5 branch, and using ClassBodyEvaluator. I wonder if downgrading to 2.5 is a short term fix? Incidentally, I personally like using the ClassBodyEvaluator for what you're doing because you don't have to name classes, and more importantly you can force a particular superclass / interface. Best wishes, Neil |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #17 on:
2011-11-13 07:28:17 » |
|
Sorry, my code is currently more similar to scrambled eggs than to actual code. I basically hackingly merged two different test programs. Go figure... T_T It's easy to reproduce though. Just create two java files, with the first one containing a method with a while(true)-loop with a break; to exit it. Then load both of them with a JavaSourceClassLoader (the while-break one first) and bam! JaninoRuntimeException!  |
There is no god.
|
|
|
avm1979
Full Member Posts: 157 Medals: 10
|
|
«
Reply #18 on:
2011-11-13 10:39:18 » |
|
Thanks for verifying, and tracking that down! Thing is, it's been over a year since 2.6.1 - wonder if 2.6.2 is going to get released at all. Ah, well, keeping my fingers crossed. ... Just create two java files, with the first one containing a method with a while(true)-loop with a break; to exit it. Then load both of them with a JavaSourceClassLoader (the while-break one first) and bam! JaninoRuntimeException! Yep, that's all it takes. Kind of surprised it took this long for someone to run into that, tbh. +1 Janino. -1 Groovy (pile of junk).
I started out using Groovy for scripting in Starfarer and had all sorts of trouble, from compilation speed to code compiling and then dying at runtime. It was all resolvable, more or less, but added up to being a huge pain. Janino, on the other hand, worked very smoothly. As an added bonus, you can use your Java IDE of choice for the scripts. I've set up my projects like this: a core project, an api project, and a scripts project. Both core and scripts depend on api, but core does not depend on scripts. That way, you get all the compile-as-you-type support from the IDE - but at runtime, since core doesn't depend on the scripts project, it doesn't see the IDE-compiled class files and can use Janino to compile them on startup. Works like a charm, and forces a clean separation of what's accessible to scripts vs not. |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #19 on:
2011-11-13 18:15:44 » |
|
Damn, the last new version was indeed released in June 2010... >_< Stupid world! I read on the front page that you can make it use JavaCompiler instead. JANINO can be configured to use the javax.tools.JavaCompiler API (available since JDK 1.6), which removes the Java 5-related limitations. Nothing anywhere on HOW to actually do that though. -_- I suppose it would require a JDK on the computer, but it would at least be a workaround until the next version is released, whenever that happens. |
There is no god.
|
|
|
avm1979
Full Member Posts: 157 Medals: 10
|
|
«
Reply #20 on:
2011-11-13 18:39:55 » |
|
Yeah, that requires the JDK. Which is a no-go for me because of the license - you can't redistribute it. Ahhh. Very frustrating. Still, can be coded around by just not using break/continue... some people say that's good coding style, anyway, though I'm not one of them  |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #21 on:
2011-11-13 19:02:26 » |
|
I got the JavaCompiler version working... Commented out lines are the equivalent Janino compiler code. rcl is my restricted class loader. 1
2
3
4
5
6
7
8
| import org.codehaus.commons.compiler.jdk.JavaSourceClassLoader;
...
JavaSourceClassLoader loader = new JavaSourceClassLoader(rcl);
loader.setSourcePath(new File[]{new File("").getAbsoluteFile()}); |
Yeah, that requires the JDK. Which is a no-go for me because of the license - you can't redistribute it. Ahhh. Very frustrating. Still, can be coded around by just not using break/continue... some people say that's good coding style, anyway, though I'm not one of them What's wrong with "Please download the Official Java JDK from Java.com to be able to run this game."? You could even have the game download it for you from Java.com? |
There is no god.
|
|
|
Riven
« League of Dukes » JGO Kernel Posts: 5867 Medals: 255
Hand over your head.
|
|
«
Reply #22 on:
2011-11-13 19:10:11 » |
|
JDK is not available on java.com
|
Hi, appreciate more people! Σ ♥ = ¾ Learn how to award medals... and work your way up the social rankings |
|
|
ra4king
JGO Kernel Posts: 3155 Medals: 196
I'm the King!
|
|
«
Reply #23 on:
2011-11-13 19:13:33 » |
|
|
|
|
|
sproingie
JGO Strike Force Posts: 894 Medals: 55
|
|
«
Reply #24 on:
2011-11-13 19:13:44 » |
|
You may not need the whole JDK. If JSR199 is available as a redistributable download, the eclipse compiler implements the javax.tools.Compiler interface too, and that's bound to have less bugs than Janino.
|
|
|
|
|
avm1979
Full Member Posts: 157 Medals: 10
|
|
«
Reply #25 on:
2011-11-13 19:46:33 » |
|
What's wrong with "Please download the Official Java JDK from Java.com to be able to run this game."? You could even have the game download it for you from Java.com?
Call me crazy, but I prefer to have the players run the game using the same JRE that it's been tested against. Besides, stuff should "just work" as much as possible - a manual step like that means someone will inevitably screw it up and need help, or get mad because the JDK installer installed crapware on their computer (which it does). Everyone is happier when the player doesn't need to do anything special to get the game running You may not need the whole JDK. If JSR199 is available as a redistributable download, the eclipse compiler implements the javax.tools.Compiler interface too, and that's bound to have less bugs than Janino.
Hmm, that may be worth looking into. Thanks for the suggestion! |
|
|
|
Mads
JGO Ninja Posts: 674 Medals: 16
Directly directional
|
|
«
Reply #26 on:
2011-11-13 22:50:26 » |
|
Here's a crazy thought; What about just using java, as with your original intent?  You'll get plenty of speed, and features won't be hard. You'll get to call all of your objects directly, and it supports all kinds of fun things. Does it need to be accessible by players? Read/write, kind of? |
|
|
|
theagentd
JGO Wizard Posts: 1392 Medals: 88
|
|
«
Reply #27 on:
2011-11-13 22:54:31 » |
|
The scripts are to be part of game maps (it's a strategy game) like unit stat calculation, on-hit scripts, campaign triggers, e.t.c, so in theory I could just force people to download JDK for the map editor and keep the compiled Java scripts in the map file, avoiding any compilation during map loading.
|
There is no god.
|
|
|
sproingie
JGO Strike Force Posts: 894 Medals: 55
|
|
«
Reply #28 on:
2011-11-13 23:01:31 » |
|
I don't think making modders download a full JDK is all that much of a stretch. Players certainly shouldn't have to.
|
|
|
|
|
Mads
JGO Ninja Posts: 674 Medals: 16
Directly directional
|
|
«
Reply #29 on:
2011-11-13 23:02:25 » |
|
The scripts are to be part of game maps (it's a strategy game) like unit stat calculation, on-hit scripts, campaign triggers, e.t.c, so in theory I could just force people to download JDK for the map editor and keep the compiled Java scripts in the map file, avoiding any compilation during map loading.
Why are they scripts? Do you want the player to change them? Do you have people writing these, that can't program java? If no, I see no point in making them scripts. I would just use the js-script-manager, and do this: 1
| ScriptManager.getScriptManager().invokeWithFailTest(o.getScriptString(), player) |
..and you could script it like 1
2
3
| function doShizzle(player) {
player.getBackpack().addItem(Items.HEAVY_ASS_SWORD);
} |
I mean.. just make it easy for yourself I don't think performance is going to be a problem this way. |
|
|
|
|
