Mobile phone specs vary widely. I had a Siemens S55 a couple of years back and struggled to write a 2D scroller without using the siemens specific class library. I had to prerender the entire level. Also transparency was not part of the standard image spec. Using the siemens specific libraries was the only option for decent performance. That is why phone games are tailered for each family of phone.
Some of the current phones are a lot faster. My Sony Ericsson 750i fairly rips along in comparison and also supports Java3D (although this isn't the same as J2SE Java3D). Simple 3D is possible on that. Thus things are getting a bit more standardised, so hopefully less porting is the way of the future.
Writing a custom (i.e. software renderer) is almost certainly a non-starter. The processor is only 200 to 400MHz at best and in most cases more like 20MHz, while the screen sizes are getting bigger. Maybe simple wire-frame graphics (BBC Elite style) would be ok. (Edit: Maybe a quake-like 3D implementation at QVGA resolution might be doable at 400MHz, provided the java implementation includes hotspot)
Another problem is the limitation of phone buttons. Quite a few phones can only see a single button press at a time, which makes real time shooter games a touch tricky.
Note that you need to download the toolkit for the phone you are targeting and even then, performance on the PC emulator is normally not representative of performance on the phone. Also note that the last time I played with J2ME was a couple of years back, so I'm not really current on it. In particular I haven't actually tried the Java3D on my 750i.
/Edit: Better 3D graphics is coming. If you've not already read it, look at this thread. Nice link to website in about 3rd post down too.http://www.java-gaming.org/forums/index.php?topic=12644.0