Java-Gaming.org    
Featured games (81)
games approved by the League of Dukes
Games in Showcase (483)
Games in Android Showcase (110)
games submitted by our members
Games in WIP (550)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: 1 [2]
  ignore  |  Print  
  Java's built in Scripting engine  (Read 6659 times)
0 Members and 1 Guest are viewing this topic.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 781
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #30 - Posted 2013-11-15 08:12:47 »

Sure it does:  while(true) is an error if you made this choice.  and so is list.add(...).

(EDIT: and I'm also talking about looking at the AST level prior to the lowering to bytecodes)
Well, yeah, if you made the choice to disallow all loops and most method calls all together, then yes, you're safe persecutioncomplex

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Roquen
« Reply #31 - Posted 2013-11-15 09:10:49 »

The goal of scripting is to allow building of behaviors and modification of game state data and not general computing.  I think what I'm suggesting has a fair amount of merit and is more flexible that what you'll see in most commercial games that provide an end-user scripting language.  You're allowing the janino (or javac) folks do the heavy lifting of conversion of source to AST and lowering to bytecodes.  The JVM folks are taking care of converting bytecodes to native.  You only have to write a custom classloader which only allows permitted classes to the scripter and an AST visitor to handles any weaving and subseting of java.  This should be fairly easy to keep up-to-date with any changes the janino/javac folks do.  On loops it seems a reasonable thing to disallow esp. if the scripts are running server side and the SDK provides some iterators such as
entitiesWithinRadius(...)
.

If you have these basics locked-down and working rock solid you could get fancy and add in work-arounds for some of these limitations by providing SDK calls.
Offline CommanderKeith
« Reply #32 - Posted 2013-11-15 17:10:14 »

I downloaded eclipse and the AST plugin,
followed the instructions to get AST working in my program independently of eclipse:
http://www.programcreek.com/2011/01/a-complete-standalone-example-of-astparser/
Managed to analyse some code using the org.eclipse.jdt.core.dom.ASTVisitor class. There are heaps of 'visit...' methods which can be used to detect things, the javadocs are here:
http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FASTVisitor.html
If I analyse this simple file:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
package eclipseast;
public class TestFile {
   public TestFile(){
      // a constructor
  }
   public boolean someMethod(){
      TestFile testFile = new TestFile();
      while (true){
         Object obj = new Object();
         break;
      }
      String str = "hi";
      return str.startsWith("hi there");
   }
}

It produces this output:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
Line 1: ASTNode of type CompilationUnit
Line 1: ASTNode of type PackageDeclaration
Line 1: ASTNode of type SimpleName named 'eclipseast'
Line 2: ASTNode of type TypeDeclaration
Line 2: ASTNode of type Modifier
Line 2: ASTNode of type SimpleName named 'TestFile'
Line 3: ASTNode of type MethodDeclaration
Line 3: ASTNode of type Modifier
Line 3: ASTNode of type SimpleName named 'TestFile'
Line 3: ASTNode of type Block
Line 6: ASTNode of type MethodDeclaration
Line 6: ASTNode of type Modifier
Line 6: ASTNode of type PrimitiveType named 'boolean'
Line 6: ASTNode of type SimpleName named 'someMethod'
Line 6: ASTNode of type Block
Line 7: ASTNode of type VariableDeclarationStatement
Line 7: ASTNode of type SimpleType named 'TestFile'
Line 7: ASTNode of type SimpleName named 'TestFile'
Line 7: ASTNode of type VariableDeclarationFragment named 'testFile'
Line 7: ASTNode of type SimpleName named 'testFile'
Line 7: ASTNode of type ClassInstanceCreation named 'TestFile'
Line 7: ASTNode of type SimpleType named 'TestFile'
Line 7: ASTNode of type SimpleName named 'TestFile'
Line 8: ASTNode of type WhileStatement
Line 8: ASTNode of type BooleanLiteral
Line 8: ASTNode of type Block
Line 9: ASTNode of type VariableDeclarationStatement
Line 9: ASTNode of type SimpleType named 'Object'
Line 9: ASTNode of type SimpleName named 'Object'
Line 9: ASTNode of type VariableDeclarationFragment named 'obj'
Line 9: ASTNode of type SimpleName named 'obj'
Line 9: ASTNode of type ClassInstanceCreation named 'Object'
Line 9: ASTNode of type SimpleType named 'Object'
Line 9: ASTNode of type SimpleName named 'Object'
Line 10: ASTNode of type BreakStatement
Line 12: ASTNode of type VariableDeclarationStatement
Line 12: ASTNode of type SimpleType named 'String'
Line 12: ASTNode of type SimpleName named 'String'
Line 12: ASTNode of type VariableDeclarationFragment named 'str'
Line 12: ASTNode of type SimpleName named 'str'
Line 12: ASTNode of type StringLiteral
Line 13: ASTNode of type ReturnStatement
Line 13: ASTNode of type MethodInvocation named 'startsWith'
Line 13: ASTNode of type SimpleName named 'str'
Line 13: ASTNode of type SimpleName named 'startsWith'
Line 13: ASTNode of type StringLiteral


The zip file with 2 simple java source files and the Eclipse EDT jar files that produce the output above is here if anyone wants to mess with it: https://dl.dropboxusercontent.com/u/50479250/EclipseAST.zip

I think it's feasible for me to weave time-checking methods into the start of all loop iterations (WhileStatement, ForStatement, ...), method calls (MethodInvocation) and others which allows me to terminate the 3rd party script in case it's carrying on for too long.

I could restrict access to all methods and classes except some that will be allowed, for example anything to do with game state or the basic classes like String, ArrayList and others. This can be done by querying the MethodInvocation or StringLiteral or other relevant ASTNode about what class of object is being created/having its method called.

But about controlling the problem of memory over-allocation, I don't think that I can easily restrict it by looking at the source code using this AST method. I mean how can I know that String[] stringArray = new String[Math.pow(100000,10000)]; is going to make a humongous String array just by looking at the source? I'm wondering if you guys know another method of monitoring memory allocation, perhaps using ASM and bytecode?

Cheers,
Keith

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Online Riven
« League of Dukes »

JGO Overlord


Medals: 781
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #33 - Posted 2013-11-15 17:33:07 »

With ASM you can alter the bytecode to first DUP the value at the top of the stack, pass it to another method to verify the arraylength is 'sane' prior to executing the NEWARRAY/ANEWARRAY opcode. That way your sanity checks happen at runtime - as that's the only time mallicious input can be detected.

As for Roquen's Iterable strategy, that provides a loophole where you can nest enhanced-for loops into oblivion as a means to create a near-infinte loop, which can be used to stage the aformentioned attacks, while all by themselves creating tonnes of Iterator instances that are strongly reachable.

Bytecode manipulation through ASM is just as easy as messing about with AST, whilst being way more powerful and effective, and less restrictive to the scripter. Let them have their custom loops Smiley

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline CommanderKeith
« Reply #34 - Posted 2013-11-15 17:42:47 »

Good idea, thanks! I'll give it a go and report back.

Offline Danny02
« Reply #35 - Posted 2013-11-15 19:36:13 »

Am I the only one thinking about the halting problem here?
Or does it not apply, because we don't want to prove anything and only set restrictions?
Online Riven
« League of Dukes »

JGO Overlord


Medals: 781
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #36 - Posted 2013-11-15 20:24:17 »

That's why you set a limit on (conditional) jumps through bytecode transformation.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Danny02
« Reply #37 - Posted 2013-11-15 20:28:04 »

what about recursion, is that included(not only tailrec)?
Online Riven
« League of Dukes »

JGO Overlord


Medals: 781
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #38 - Posted 2013-11-15 20:42:28 »

Please read the thread Pointing

create a callback prior to every new, newarray, anewarray, multianewarray, invoke*, goto, goto_w, if*, if_*, jsr, ret and athrow

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Roquen
« Reply #39 - Posted 2013-11-15 20:48:54 »

Quote
I downloaded eclipse and the AST plugin...
Seems to be a minor bit of mis-communication.  By AST I meant whatever AST/Visitor functionality the choosen compiler based provides and not any specific library...so janino has this in it, so does javac and (yes) eclipse.

Quote
But about controlling the problem of memory over-allocation, I don't think that I can easily restrict it by looking at the source code using this AST method.
I'm suggesting that the user not be allocate memory for themselves...ever.  They can only get an instance of something from code you provide.  That's the only way you can insure a bound on allocations.  Related to that is why I'm suggestion you not give them access to any standard java classes, beyond those pretty much required: like primitive wrappers and String.  Every class you allow them direct access to needs to be inspected for any potential holes and you need to redo that work any time you upgrade the JVM.  Recall what I said above...you don't need to provide general computation framework...just things they need to be able to define behaviors and modify accessible game states.

Quote
what about recursion, is that included(not only tailrec)?
I doesn't matter what the script does..if it takes too long it's forced to stop.  No code analysis is needed.

Quote
As for Roquen's Iterable strategy, that provides a loophole...
Good point.  Back to weaving in a checks at top of loops.  The reason I thought it would be good to avoid this is so the server could run fewer serializing instructions (reading the counter).

Quote
Bytecode manipulation through ASM is just as easy as messing about with AST...
In the cases we're currently talking about the choice doesn't really matter too much.

Quote
whilst being way more powerful and effective,
Pfffff...
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Online Riven
« League of Dukes »

JGO Overlord


Medals: 781
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #40 - Posted 2013-11-15 21:27:04 »

Quote
Bytecode manipulation through ASM is just as easy as messing about with AST...
In the cases we're currently talking about the choice doesn't really matter too much.

Quote
whilst being way more powerful and effective,
Pfffff...
The issue is that with an AST/analyzing approach you have to severely limit the functionality you expose to the scripter, to guarantee it to run within finite resources. With ASM/rewriting you can analyze the behaviour of the script at runtime, halting/interrupting the script when you deem it appropriate, while retaining all features that should reasonably be available to the scripter.

That's all, folks, because 'Pfffff...' is a tad hard to expand upon Smiley

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline Roquen
« Reply #41 - Posted 2013-11-16 07:33:01 »

I don't see the basis of this statement (hence the Pffff)...anything you can do in one you can do in the other.  The disadvantage of ASM is that you've already thrown away high-level structure information and you're working with lower level constructions.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 781
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #42 - Posted 2013-11-16 10:56:28 »

With AST you can't determine whether
arr = new int[n]
is reasonable, while with ASM you can.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online princec

JGO Kernel


Medals: 362
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #43 - Posted 2013-11-16 11:45:01 »

If only Java had some sort of resource management thing built in to it to manage allocation of system resources... and application segregation. Then you could just say, "you're allowed n files open, n megabytes in a heap of your own, n threads running at NORM_PRIORITY-1 or less, which are each allowed to run for n seconds before they're forcibly terminated" and that sort of thing.

Cas Smiley

Offline Roquen
« Reply #44 - Posted 2013-11-16 12:37:40 »

With AST you can't determine whether
arr = new int[n]
is reasonable, while with ASM you can.
Huh?  But that's beside the point from my perspective.  I'd not allow any direct memory allocations by scripting if the target is server side. 
Online Riven
« League of Dukes »

JGO Overlord


Medals: 781
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #45 - Posted 2013-11-16 13:33:36 »

That's the core of our miscommunication. You argue that within all the restrictions you impose, both solutions are equal. That is correct, but my solution doesn't require these restrictions.

To extrapolate your solution: if you minimize the exposed functionality to nothing, nothing bad can happen.

To extrapolate my solution: if you validate every execution of each instruction, nothing bad can happen.

For a general purpose solution, I'd pick the solution imposing the least amount of restrictions, as it directly affects the way you can express ideas through code, and therefore productivity.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline CommanderKeith
« Reply #46 - Posted 2013-11-16 18:43:32 »

Hi,
I finally got ASM working after a lot of reading about how JVM byte codes and ClassLoaders work.

I made some code that reads an existing class's byte codes, modifies the byte code by adding some extra characters onto a String field within the class's method's body, then calls that method and prints the String returned which shows that the byte code changes worked and the String is different to the original. If anyone wants to mess with it, here it is: https://dl.dropboxusercontent.com/u/50479250/ASMTest.zip

Thanks for pointing out the opcodes that I should look into Riven:
Quote
new, newarray, anewarray, multianewarray, invoke*, goto, goto_w, if*, if_*, jsr, ret and athrow
I don't understand bytecodes and compilers well enough yet, but once I've read some more about these I'll probably have a few questions about how to calculate the amount of memory being allocated with each opcode, and how to chase the allocations made in other objects' methods if they're called.

I'm really looking forward to seeing this working. It'll be cool to have safe and secure client-programmable scripts that run on the server. Once the code's reasonably secure, I'll host a web page where we can upload and execute malicious java code on the server to see if these methods are effective

For anyone wanting to learn about ASM, these were good resources:
http://asm.ow2.org/doc/faq.html
http://download.forge.objectweb.org/asm/asm4-guide.pdf

Thanks for the advice and help guys.

Offline Roquen
« Reply #47 - Posted 2013-11-17 19:26:16 »

@Riven: Neither of us are currently discussing difficult transforms, so the choice between bytecode vs. AST manipulation is not important if your equally familiar with both.  Otherwise it's what you've more experience with and for people with none in either I'd expect AST to be easier.  YMMV.

On the array allocation example for instance (since I provided no details) I don't see that replacing the length expression with a wrapped expression (really an rvalue) is impossibly difficult...it's the same difficulty as adding a dup and a method call as far as I'm concerned.  But enough of that subject.

To extrapolate your solution: if you minimize the exposed functionality to nothing, nothing bad can happen.
My suggestion is to provide DSL/Scripting functionality to scripters rather than general purpose programming constructs which are potentially problematic for running on a server.  Let's keep in mind that people that end up scripting stuff for games frequently have little to no programming experience (not that I'm suggesting to cater to these folks.)  If initially the scripting language itself is too restrictive it's always possible to attempt to address that at a future point assuming it seems somehow worthwhile.  The engineering cost of this solution is low.

To extrapolate my solution: if you validate every execution of each instruction, nothing bad can happen.
Except it not as easy so you're presenting.  You must not allow the engine nor any other classes hold any references to the allocated memory to insure it doesn't escape the script's entry point.  The tricky part here is the scripter's own classes via static fields.  Additionally you need to track the (rough) amount of memory allocated from script entry and toss an exception if you goes beyond some threshold.  Sure this isn't hard...it simply more up-front work to support a feature a scripter shouldn't really need in the first place.

EDIT: Oh..and I left out any instance fields of scripter objects which the engine might hold cannot hold a reference to any other scripter object or engine provided container/array.
Offline Roquen
« Reply #48 - Posted 2013-11-20 16:46:02 »

FYI: (three letters there monster) - the javac AST framework seems to be read-only...unless I'm missing something.  Boo.

Posted a couple of seconds too early: com.sun.tools.javac.tree.JCTree
Pages: 1 [2]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

CopyableCougar4 (15 views)
2014-08-22 19:31:30

atombrot (28 views)
2014-08-19 09:29:53

Tekkerue (25 views)
2014-08-16 06:45:27

Tekkerue (23 views)
2014-08-16 06:22:17

Tekkerue (15 views)
2014-08-16 06:20:21

Tekkerue (22 views)
2014-08-16 06:12:11

Rayexar (61 views)
2014-08-11 02:49:23

BurntPizza (39 views)
2014-08-09 21:09:32

BurntPizza (31 views)
2014-08-08 02:01:56

Norakomi (38 views)
2014-08-06 19:49:38
List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06

List of Learning Resources
by SilverTiger
2014-07-31 11:54:12

HotSpot Options
by dleskov
2014-07-08 01:59:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!