I think Project Sumatra and also Graal and Truffle
will be very important for the future of the JVM. Oracle is not alone either, AMD has big plans
for Java performance. For now though, the problem is two-fold:
a) We won't have a useful implementation until Java 9. That's 2016, 2+ years from now. Java 8 will be the major feature release (lambdas) and Java 9 the major performance one (HSA plus Arrays2.0/PackedObjects/Structs/whatever).
b) I personally don't think automatic offloading to GPU is going to fly on current PC architectures. The non uniform nature of memory, i.e. separate virtual address spaces for CPU and GPU, is very hard to program on and even harder to optimize for. Even with manual effort, very few types of workload are actually accelerated by taking advantage of the GPU, the transfers between CPU and GPU memory are simply too costly. Hacks like CUDA 6's Unified Memory may simplify programming, but do nothing about actual performance. Next-gen APUs (e.g. AMD's Kaveri) will fix this problem and only then can I see general purpose Java code easily optimizeable on there. So, we're again talking about a few (many?) years for such architectures to grab enough market share.
In any case, we'll have to wait.