I was wondering what the current thinking is on 3D engines' design might be these days.
I'd say there are many dimensions to this and I'll answer from the perspective of what I'd build..
For outdoor environments I'd focus on octree partitioning with an emphasis on texture / geometry streaming and further tessellation for enhanced detail. For indoors / non-destructive you simply can't do better than portal/bsp/pvs to my knowledge. One could still support some level of streaming & dynamically loading BSP and other data for expansive indoors environments.
Technologically speaking especially for outdoor and streaming data parallelism is crucial. This means supporting multi-threaded GL and CPU side architecture. From a Java perspective I personally would really like to start using the Disruptor on the CPU side of things which would handle anything routing between the disk and the network to threads streaming through to GL. I am bit sad though that the Disruptor doesn't work on Android due to its reliance on sun.misc.Unsafe. If Android is a target and it should be this means sticking to OpenGL ES for the time being though that could be different a couple years from now and things may properly converge to just OpenGL across all mobile SoC/GPU vendors. This means though supporting OpenGL ES 3.0 and the GL threading API and while one is at it all the other goodness 3.0 brings.
OpenGL AZDO is important of course. There are many aspects there that support texture streaming / binding concerns. I'd figure out the limitations of MultiDrawIndirect. I really wished that the first Android Extension Pack covered AZDO extensions. I'll be able get started with them on the K1 though since it supports OpenGL 4.x too. Though I'll have to think about how the majority of the engine work must remain compatible with GLES 3.0 right now.
Compute shaders and OpenCL is very important. Particle engines will all be compute based. In fact the first demo I'm going to port to Java is this one:http://docs.nvidia.com/gameworks/index.html#gameworkslibrary/graphicssamples/opengl_samples/computeparticlessample.htm
For other modern GL demos check out all of them:https://developer.nvidia.com/gameworks-opengl-samples
To have any chance at maintaining an engine now and into the future modularity must be a concern from the get go or go the way of all past Java engines. Component architectures (CA) are key in this direction IMHO. If you don't think so, well, I can't help you there much.
A CA is flexible to support structuring both the engine subsystems at a high level that is compatible with data parallelism in addition to other aspects like entity systems. It also happens that it supports various rendering paths and hardware diversification concerns which is quite important for long term evolution of an engine. It is what is going to make relatively painless my efforts to support OpenGL ES 3.0 for the majority of Android devices and have a GL 4.x path compatible path with the K1. It also makes tooling possible hence the nice user experience Cas also mentions when he speaks favorably of Unity.
I'm so excited that I get to start down parts of the rabbit hole above tomorrow when my Shield Tabby arrives..
The only real shortcut is picking a specific visual style and focus on gameplay...
Gameplay is king at the end of the day. From an indie perspective one could consider procedural content generation and have great gameplay.
The first example that pops to mind in regard to picking a specific visual style and procedural content is Love:http://www.quelsolaar.com/love/
I never have gotten around to checking it out though so, I'm not sure about game play.
What I think would really benefit the larger Java game dev community is as many folks as possible getting their hands some of the leading tech today (Right at this moment on Android that is the Tegra Shield Tabby, desktop there are various high end GPUs to pick) and start building new engines that may not be compatible with the wider ecosystem. OpenGL ES 3.0 should be the absolute minimum supported for new engine development right now and experiments with OpenGL 4.x should occur on mobile and if you are on the desktop only ignore anything that isn't the latest GL 4.x point releases.