Another game using java as scripting language is Chrome (http://www.chromethegame.com
, 3D FPS similar to UT). Their java code is totally ugly, but it works.
I would certainly advise against creating your own language+runtime from the scratch. There is a possibility of creating your own language and running it on jvm - this way you will have solid runtime and need to play only with compiler.
Big question is how much customizability you want to expose for end user. For example, in NWN Bioware has created system where everybody and their dog were supposed to be writing scripts for modules. They have decided to stick to most basic C subset (no pointers etc) for scripting language and it still was too high bar for majority of people. Bioware ended with creating 'wizards' which would take few inputs and generate code for you. I hope you do not plan to target same audience - in such case, ONLY thing which is important is ease of use for non-programmers (which is impossible anyway, because if you are non-programmer, you cannot think in algorithms, regardless of how easy language is).
For mod community (like UT/Quake), you can basically use anything - they will even learn assembly if it is needed
I think that you know most pro-java arguments yourself. I would like to point my personal worries, why it is NOT so perfect for gaming scripting language (in no particular order):
1) It is overly verbose as far as standard library is concerned. While it helps in big system programming, in scripts you may prefer function points per line rather than fully self-documenting readable code.
2) It requires a lot of context to do anything. Helper methods, setters, constructors, a lot of writing to get simpliest things done. Please look at http://www-users.cs.york.ac.uk/~susan/joke/foot.htm
Java entry to see what I mean here. Again, for enterprise-level, extensible systems it is beneficial, for game scripts it can be tedious.
3) For mixed language engine it is non-perfect. You need to non-OO at languages boundary to use JNI - you will need either to mirror most of data to java (which will cause synchronization problems and memory bloat) or ask for everything through non-OO interface which means you can as well use any procedural language.
4) Java is designed to be very generic. Maybe for your game, some other progamming paradigm would be more useful ? Best example for me is Stackless Python used in Eve Online. With very light threads/continuations, they just run separate program for every object in world, being able to write code which just pauses for a moment if needed - not acceptable in shared thread environment (doing it with java/C threads is not possible due to amount of stack memory needed per thread/OS limits).
5) Rapid prototyping isn't really a java strength. You can try to do tricks with class reloading, but migrating state can be hard or impossible. Language/environment in which you could edit script while game is live and see effects at very instance can be a major productivity gain.
1,2 and 5 can be solved by using some kind of other language built on top of jvm. Maybe beanshell (http://www.beanshell.org/
) ? It is opensource and with a bit of tailoring you should be able to add few tricks for it (like prototyping new behaviour on single instance of object). For routines which will turn out of be performance hogs, you can always port them to java and call from beanshell (which should be mostly trivial).
3 is not a problem if you will write game in pure java. For mixed language, interfacing will be always a problem - but AFAIK, some other languages have more explicit constructs for overcoming the barrier.
4 - Stackless Python rulez
I'm a bit afraid about performance of heavy code (like pathfinding/AI), but for object behaviour it seems like a really reasonable solution.
To reassume: If you will decide on core engine in java, use jvm to run scripting language - I would propose beanshell (second choice would be JPython) plus java helper classes, all of it having access to internal engine classes if needed. If you want to have engine in C++, consider Stackless Python. C++ and jvm as scripting engine - be prepared to see that your java code will be just C (not C++) as far as programming style is concerned.
IMHO, AFAIK, my 0.02$ and all other disclaimers follow...