Java's built in Scripting engine

[jmguillemette]

Java is a great language .. you can work your whole career in it and still not know all of it.
Case in point... I tripped across this tonight and had no idea it even existed...

Anyone ever try to use it to script events in their game?

http://docs.oracle.com/javase/6/docs/technotes/guides/scripting/programmer_guide/#helloworld


edited - corrected typos.. too late and not enough caffeine

[jmguillemette]

quick follow up to my original post..

the scripting is pretty awesome so far...

Not sure on performance under load.. but what ive managed to do create a script executor and then externalize some of the flow / event scripting of my game.

here is a simple script that changes the current scene in the game after a 5 second delay.
(note: my script executor ensures the script is running in a separate thread)

1   2   3   4   5   6   7   8   9

importPackage(java.lang); importPackage(com.windpoweredgames.mocha.script); importPackage(com.windpoweredgames.game.aftermath.scene); importPackage(com.java); println("executing script - splash.js"); var scene2 = new SplashScene2(); Thread.sleep(5000); gameEngine.queueScene(scene2);

I'm definitely going to play with it more and report back on impressions.

[mwarner]

I've often experimented with scripting in my game but whenever I do it I bind my own classes through the script engine:

1   2

ScriptEngine engine = factory.getEngineByName("JavaScript"); engine.put("Name", class);

So if I put in for example java's System class

1	engine.put("sys", System);

In javascript you could do

1	sys.out.println(".....");

It works same, but is just a different way
Good luck with it, I always have found it very powerful!

Games published by our own members! Check 'em out!

[Riven]

Beware, it's an outdated version of Rhino...

[mwarner]

Beware, it's an outdated version of Rhino...

Would it be better to download the newer version and just use the jar?
i.e. https://developer.mozilla.org/en-US/docs/Rhino/Download_Rhino?redirectlocale=en-US&redirectslug=RhinoDownload

[CodeHead]

I have plans to integrate scripting in the engine I'm working on now, but have bigger fish to fry with the code base at the moment. I prefer to use the standalone version of Rhino distributed by Mozilla. Not only is it a newer distribution, as Riven pointed out, it also seems to be more feature rich. I tried out the internal scripting engine over the weekend and was disappointed in the lack of documentation. I couldn't find a clear cut way to do things such as class shuttering (locking down what Java classes/methods are visible to a script) which may not matter in your scenario, but can be important if you plan to allow users to write custom scripts. There is a little more setup to do with Rhino, but it's nothing overly complex.

That being said, I've got a few work projects that utilize scripting to control application logic and found Rhino to be rock solid in real world use cases. I also recommend considering the approach mentioned by mwarner of exporting only classes that you want to be accessible to the engine. I usually take it one step further and write wrapper classes for the objects that I want to export instead of exporting the core objects directly. It has the advantage of added flexibility in respect to what the scripting environment can access.

Would it be better to download the newer version and just use the jar?
i.e. https://developer.mozilla.org/en-US/docs/Rhino/Download_Rhino?redirectlocale=en-US&redirectslug=RhinoDownload

Yes. Version 1.7 r4 should be the newest. Just include the js.jar file in your project, and you're all set.

[jmguillemette]

thanks for the tip.. ill check it out

[CommanderKeith]

Nashorn is the very latest Rhino. It'll be included in java 8:
http://openjdk.java.net/projects/nashorn/

And a funny little javascript + javafx example that I haven't tried but look interesting:
http://justmy2bits.com/2013/09/08/javafx-with-nashorn-canvas-example/

[Abuse]

Nashorn is the very latest Rhino. It'll be included in java 8:
http://openjdk.java.net/projects/nashorn/

And a funny little javascript + javafx example that I haven't tried but look interesting:
http://justmy2bits.com/2013/09/08/javafx-with-nashorn-canvas-example/

I thought the whole purpose of scripting was the reduce complexity for the script writer?

That example is line-for-line almost exactly the same as the Java equivalent.

[CommanderKeith]

Lol that's true. I'm a javascript n00b so I actually appreciate the java-styled javascript much better.

The benefit of javascript for me is being able to easily block access by untrusted users to the JVM and server computer, but still allow them to run scripts on my server.

I look forward to being able to take javascript code typed by someone in a browser and execute it on my server without fear that it can trash my Tomcat webserver JVM. For example I could use javascript to call java methods on my server which could fetch data from the database, and manipulate it on the server in javascript and send back the results to the browser.
Or I could allow players in a game to program their own AI in javascript and then execute it on my server in-game.

Using janino (http://docs.codehaus.org/display/JANINO/Home) and real java rather than javascript would be better but I don't know how to properly restrict access to the sensitive libraries in the JVM from malicious users.
I've tried to learn about java's security policies but everytime I try it seems far too complicated.
I also remember how Riven ran the tiny code program competition and challenged you to hack it and somehow you did. Since Riven overlooked whatever hole you found in his security, I've decided to not even try.

But with Nashorn/Rhino/Javascript I can easily block out the dangerous bits of java so there's no calls to System.exit() etc.

Games published by our own members! Check 'em out!

[Roquen]

Your classloader can deal with restrictions.

[SHC]

BeanShell2!! You can simply use the java code in it or JS style code, it can run anything!

1   2   3   4

public int add(int a, int b) {     return a+b; }

Is essentially the same as

1   2   3   4

add(a, b) {     return a+b; }

And this is how you create the interpreter.

1   2   3   4   5

import bsh.Interpreter; Interpreter i = new Interpreter(); i.source("somefile.bsh");

Hope this helps.

[CommanderKeith]

BeanShell is much more popular than Janino for some reason, but I think it's much worse since it's slower.
Janino converts code into real java classes that run as fast as any other java code. BeanShell is interpreted and slow.

Your classloader can deal with restrictions.

I don't know how to restrict classloaders for that purpose, I assume it's through using SecurityManager which I'm yet to figure out. Also I think that Apache Tomcat (and most java webservers) do their own magic with classloaders which might make it even harder.

Cheers,
keith

[Roquen]

You write a custom classloader (based on a janino one in this case) which only allows access to classes you wish to expose.

[SHC]

@CommanderKeith

I use BeanShell because I want to execute the scripts directly, not compile them in which case, I would have used normal Java. Also BeanShell is not slow, it is used in David Brackeen's book Developing Games in Java and more importantly while creating a 3D software renderer, and the code performed at 70-80 fps on my old laptop with no graphics card.

[CommanderKeith]

Janino compiles and runs the script, there's no disadvantages of using Janino, only the benefit of it being faster.

About your example that ran a 3d game, I doubt the entire game was written in BeanShell or else the FPS would be very low. I'm guessing that BeanShell just called the update method and the rest was done in normal java and openGL, so the bottleneck was not the scripting, in which case it wouldn't matter what scripting engine you use.

[Riven]

I also remember how Riven ran the tiny code program competition and challenged you to hack it and somehow you did. Since Riven overlooked whatever hole you found in his security, I've decided to not even try.

Don't draw too many conclusions from that. I decided to do it the easy way at the time, by not allowing capitals in the sourcecode. This did the trick surprisingly well, except that Abuse found a class in the JRE that didn't adhere to the Java naming conventions and the method call halted the JVM. How lucky he was

[CommanderKeith]

Haha, what a novel approach!

On the topic of locking down code, Google have an interesting library that locks down javascript code run on the client by untrusted third parties: https://code.google.com/p/google-caja/

[nsigma]

Janino compiles and runs the script, there's no disadvantages of using Janino, only the benefit of it being faster.

Yes, I think @SHC is missing the point of Janino.  I use it in Praxis LIVE so that users can write custom components on-the-fly, that run at the same speed as anything built in.

Janino does have a disadvantage though, in that you're pretty much stuck with Java 1.4 source code features.  There's a few things I have planned for the future that might require things like annotations.  Now, there is a Janino interface for javac as well, that handles all the in-memory class loading, expression wrapping, etc. - the only problem is that it requires the user to have a JDK.

Hence, I spent yesterday afternoon refactoring javac and the Janino JDK interfaces into a stand-alone compiler lib.  Working so far, though a fair bit more testing to do.  Weird coincidence this thread! 

... would be better but I don't know how to properly restrict access to the sensitive libraries in the JVM from malicious users.

In many ways if we're talking about end-user scripts I'm less concerned about that (unless you're promoting distribution of course) - let them break their own stuff!  I would love to find a good way of controlling things like infinite loops, without adding loads of extra overhead / memory inconsistency.

[CommanderKeith]

Sounds like a cool piece of software you built.
It made me think of the eclipse IDE on the floy compiler, would it also be another way of doing what you want? Here's some details about it:
http://stackoverflow.com/questions/14226357/how-to-do-standalone-on-the-fly-in-memory-compilation-with-the-eclipse-compiler

... would be better but I don't know how to properly restrict access to the sensitive libraries in the JVM from malicious users.

In many ways if we're talking about end-user scripts I'm less concerned about that (unless you're promoting distribution of course) - let them break their own stuff!  I would love to find a good way of controlling things like infinite loops, without adding loads of extra overhead / memory inconsistency.

Ah I see, your application runs on the client's machine.
I want to run code on my server that the client submits, so I'd like a way to restrict what they can do. I ws reading about SecurityManagers but they seem to restrict the whole JVM. I'll have to look into Roquen's suggestion of using ClassLoaders.

About the problem of infinite loops, I was reading a while back that it can be mathematically/logically impossible to tell if a program will ever stop. But I guess by having a timer you could just terminate a thread that ran for too long.

[nsigma]

Sounds like a cool piece of software you built.

Thanks! 

It made me think of the eclipse IDE on the floy compiler, would it also be another way of doing what you want?

Possibly, except for a (*&%£* license incompatibility.

I'm not sure I could achieve quite what I want without the boilerplate that Janino provides anyway.

I want to run code on my server that the client submits, so I'd like a way to restrict what they can do. I ws reading about SecurityManagers but they seem to restrict the whole JVM. I'll have to look into Roquen's suggestion of using ClassLoaders.

Ah, I see you like playing with fire! 

The Janino ClassLoader already has the ability to provide a ProtectionDomain for classes compiled through it.  That would appear to be the way to do this, but I've never tried.

About the problem of infinite loops, I was reading a while back that it can be mathematically/logically impossible to tell if a program will ever stop. But I guess by having a timer you could just terminate a thread that ran for too long.

Exactly.  The timer issue is that you then need a second monitoring thread, which potentially brings in thread switching issues - in particular one thing I'm looking at doing at the moment is live coding audio DSP.  Extra threads could be an issue there.

The bigger thing is what to do with a thread stuck in a loop.  Call stop() on it?  That would have to be very carefully thought out if anything's shared.  Would have thought that would be even worse for you - could have more serious effects on a server!

[CommanderKeith]

Hmm, interesting problem about the thread stopping. I guess Thread.interrupt won't work in an infinite loop.
I've never used it but how about byte code injection? That seems to be a way of adding arbitrary code over the top of someone else's code. Maybe you could inject some code that checks a flag at the start of every loop to see if the thread should die??
Another thing, why is it an issue for you if your user breaks his own program by making an infinite loop?

Janino is slowly adopting more recent java language specs, see the change log: http://janino.net/changelog.html
and future directions: http://docs.codehaus.org/display/JANINO/Licensing

Great, I didn't know that Janino had security features. I tried searching about it and found this:
http://dist.codehaus.org/janino/javadoc/org/codehaus/janino/SimpleCompiler.html#setParentClassLoader(java.lang.ClassLoader, java.lang.Class[])
There's not much more information/documentation than that. I'll have to dig into the janino source to figure out how to use it.

EDIT: Sorry I missed the point about ProtectionDomains. Cool, i'll look into that too: http://dist.codehaus.org/janino/javadoc/org/codehaus/commons/compiler/AbstractJavaSourceClassLoader.ProtectionDomainFactory.html#getProtectionDomain(java.lang.String)

Cheers,
Keith

[Roquen]

On preventing run-away scripts.  An other option is to do code weaving and have the script itself bail if it taking too long.  If you've never goofed with compilers this might be too much of a time commitment.  Luckily with janino (or javac framework) you can manipulate the AST instead of asm.  The upside is that you could do other code inspection and weaving.  Say disallowing most calls of 'new' or whatever else.

Humm...I'm not familiar with the ProtectionDomain class and glancing at the source it looks like it would take longer for me to figure out than just writing a classloader.  If this notion isn't clear...shout out.

[nsigma]

Another thing, why is it an issue for you if your user breaks his own program by making an infinite loop?

Simply that it isn't very user friendly!  Praxis LIVE is a graphical patcher / dataflow environment where you can add fragments of code (Java, GLSL, etc.) to the processing graph at runtime.  The idea is that you can incrementally change and back out code as it runs.  Entering an infinite loop would completely stall the media pipeline, forcing a restart and losing some changes.  If someone writes a while(true) loop without thinking, they get what they deserve - as you implied, some infinite loops are more subtle.

I may consider a protected environment, probably as Roquen suggests using AST manipulation rather than byte-code manipulation, if I can get it to run without losing too much performance.  It's not a high-priority though.

Janino is slowly adopting more recent java language specs

Slowly is unfortunately the word though. 

Great, I didn't know that Janino had security features. I tried searching about it and found this:
http://dist.codehaus.org/janino/javadoc/org/codehaus/janino/SimpleCompiler.html#setParentClassLoader(java.lang.ClassLoader, java.lang.Class[])

In the current sources I was playing with yesterday this is marked deprecated, and is also a no-op.  JavaDoc comment says "Auxiliary classes never really worked... don't use them." 

[Abuse]

I suppose a small plug for Starsector (formerly Starfarer) would not be entirely inappropriate here, given its use of Janino in its scripting & extensive modding support.

I don't know if Alex (the game's creator) frequents these forums, but he could undoubtedly cast some valuable light upon the usability & security of using Janino in this scenario.

[Riven]

With serverside 3rd party code, you have to not only worry about cpu-cycle consuming runaway scripts. Memory consuming scripts are much more dangerous, because you basically can't gracefully recover from code like this:

1   2   3   4

String s = ""; while(true) {    s += " "; }

as random, unrelated, critical code will start throwing OutOfMemoryErrors - it will very likely pull your service down when the GC panics, and maybe even the entire server, due to excessive swapping.

With ASM, you can rewrite bytecode to intercept every new and anew instruction, and manage the allocation count your script is allowed to reach. It's not easy to make this water tight, because almost all the JRE classes expose methods that do allocations behind the scenes (like StringBuilder.append, as per the above example)


I'd propose the 3rd party code to run in a separate, bolted down JVM, which can be nuked from orbit when it misbehaves. Not very practical, but what ya gonna do. With a bit of trickery, you *could* use MappedByteBuffers to share a read-only view of your business objects. (if they are backed by buffers - where are structs when you need 'em)

[CommanderKeith]

Great tips guys, I hadn't heard of AST or ASM. Both look very useful.

On preventing run-away scripts.  An other option is to do code weaving and have the script itself bail if it taking too long.

Great idea, thanks Roquen. I'll try this approach first.

1   2   3   4

String s = ""; while(true) {    s += " "; }

Hmm, i never thought of that problem, thanks Riven. I'll put your code through AST and see if it's possible to detect the implicit string allocation.

In the current sources I was playing with yesterday this is marked deprecated, and is also a no-op.  JavaDoc comment says "Auxiliary classes never really worked... don't use them." 

Lol, that's too bad.

I suppose a small plug for Starsector (formerly Starfarer) would not be entirely inappropriate here, given its use of Janino in its scripting & extensive modding support.
I don't know if Alex (the game's creator) frequents these forums, but he could undoubtedly cast some valuable light upon the usability & security of using Janino in this scenario.

Great, I'll drop him a note on his forum/mailing list. Cheers

[Roquen]

@Riven's comment:  This is related to disallowing almost all 'new' invocations in user scripts (the almost part is to allow things like boxing).  Have all instances come from SDK calls.  Disallow StringBuilder and StringBuffer in the script classloader.  In fact disallow pretty much every class that isn't from the SDK (the SDK's classloader can handle arbitrary classes) and required for basic operation.

On code weaving: I was initially thinking of weaving in a method call (which checks for timeout and tosses an exception if needed) at the entry of all user defined methods (at least to start with...you can get clever later) and at the top of loops.  But if your inspecting for user-defined loops...you could just disallow them and only allow iterating on SDK provided collections.

[Riven]

That still doesn't quite cover:

while(true) list.add(null);

You could create a callback prior to every new, newarray, anewarray, multianewarray, invoke*, goto, goto_w, if*, if_*, jsr, ret and athrow though. That way you can be reasonably sure you can intercept a runaway script, assuming that it can only create a limited amount of instances as you limit the number of instructions that can be executed, without being at the mercy of the OS thread scheduler.

Anyway, this is getting a bit offtopic.

[Roquen]

Sure it does:  while(true) is an error if you made this choice.  and so is list.add(...).

(EDIT: and I'm also talking about looking at the AST level prior to the lowering to bytecodes)