Mac OS X 10.2.6  Java 1.4.1-client  (1GHz)
Cross product local variable speed testing
1)local double var.... avg 0.082891665 usecs
2)new................. avg 0.14235833 usecs
3)ThreadLocal..........avg 0.14793333 usecs
4)Instance field.......avg 0.06655 usecs
5)Instance field sync..avg 0.12054167 usecs
6)TempStack............avg 0.16006666 usecs
7)TempStack param......avg 0.09599167 usecs

Note that #5 is cheaper than #2,3,6 which is significantly different from your results with the 1.4.2 Intel VM

Mac OS X 10.2.6  Java 1.4.1-server  (1GHz)
Cross product local variable speed testing
1)local double var.....avg 0.08573333 usecs
2)new..................avg 0.1584 usecs
3)ThreadLocal..........avg 0.153775 usecs
4)Instance field.......avg 0.06821667 usecs
5)Instance field sync..avg 0.124858335 usecs
6)TempStack............avg 0.168125 usecs 
7)TempStack param......avg 0.10123333 usecs

Thanks for the additional data points.  A few observations

• All three client JVMs seem to preserve the oddity that using a field (4) is cheaper than using local variables (1).  I still don't understand why this is the case.
  
• The relative cost of a synchronized method (5) seems to be lower on MacOSX and much lower under Redhat as compared to my Windows results.
  
• The relative cost of new and garbage collection seems slightly lower under MacOSX but significantly higher under Redhat.  It still large enough in all cases to be a potential bottleneck in truly performance-critical code.
  
• Under MacOSX, -client and -server are not significantly different which is not surprising since my understanding was that -server is ignored under the current MacOSX JVM.

Yes, there is only a single shared library for the hotspot VM on Mac OS X.  I didn't realise at first, but the server shared library and client shared library files are both there, but as alias's to a single 'hotspot' library.

I guess the slight differences are caused by different parameter values to the same VM  (e.g. compile thresholds, size of young generation in heap etc.)

I too have no clue why a field is faster than a stack variable - weird.  If only there was a way to disassemble the native ode produced by HotSpot.

The PPC architecture uses register windowing and suchlike doesn't it from its RISC beginnings? It's pretty poorly adapted for stack-based architectures like the JVM, whereas the x86, with its paucity of general purpose registers, is much better at stack ops. So let's guess that the fields get mirrored into registers on PPC.

Cas

> *  As others have noted, under 1.4.2 -server is much faster for >floating point code than -client

For something quite fundamental as floating point performance,
just would like to know the reason behind this discrepancy
between client and server options.

As a rough benchmark, I ran some cases with my Java3D particle
tracking algorithm that involves fully double precision calcs.,
newing of dynamic primitive arrays and objects of that type only,
no synchronization anywhere, extensive polymorphic method calls
(since the cells are different kinds of polyhedron), no accounting
for gc times, and no newing of objects within loops.

The time taken for creating 1000 traces for repeated invocations without any pauses are (in secs):

client:   30.98, 31.20, 32.84, 20.98, 21.48, 20.82
server: 25.02, 10.32, 11.15, 10.65, 10.25. 10.27

Looks like after the initial warm-up period, the server is roughly
twice as fast.

What would it take to get Sun to spruce up the client so it would number crunch as fast as the server ?

Edit: Forgot to ask if anyone has compared Java vs C++ FP
performance with the JVM in server mode ? That may be very
interesting.....I have done some here but not exactly apples to
apples comparisons.

--- Field (4) faster than local variables (1) under -client
I've profiled the code using VTune which has the side benefit of allowing one to view the assembly code produced by the hotspot compiler.  I've posted the resulting assembly code for the local variable and field routines here (sorry for the strange formatting):
http://www.graphics.cornell.edu/~bjw/CPTLocalVarClient.txt
http://www.graphics.cornell.edu/~bjw/CPTFieldClient.txt
I'm not an x86 assembly expert but perhaps there is an expert out there who can analyze the differences.  One thing I noticed is that local variable code computes the results using fp registers and then copies the results using int registers while the field code uses fp registers throughout.

--- Field (4) much slower than local variables (1) under -server
This turns out to be an inlining effect.  Hotspot inlines method (1) by default but not method (4).  If I disable all inlining (using the -XX:MaxInlineSize=1 -XX:FreqInlineSize=1 flags) then the local var method slows down to 0.038 us (or just a hair faster than field).  However I could not find any parameter setting that would convince hotspot to inline the field method (4) the same way it is inlining method (1) by default.

Incidently I don't think its actually any more difficult to generate the SSE/2 instructions instead of x87 for the floating point code (in fact the SSE/2 code is simpler and probably easier to generate).  Hotspot -server does not use the SIMD parts of SSE/2, just the scalar instructions as shown in the assembly code linked below.  I think SSE/2 fp code is a feature that is likely to migrate down into the client JVM in the next version, especially if enough people request it.
http://www.graphics.cornell.edu/~bjw/CPTLocalVarServer.txt