I have the same issue on my GeForce Ti 4600, it isn't 5 seconds but the first call does take noticably longer than the rest.
Is your scenegraph filled with a lot?
Does your "system" cache - ie is the first render call still delayed when you start your Xith app a 2nd time within a short time?
My scenegraph is very small, about three primitive objects, so that can't be the issue.
On a Geforce TI4200 on the same PC the same Java Xith app doesn't take "extra seconds" for the first render call. I just noticed it on an ATI gfx card, but maybe it's a general problem...
Kev, did your profile say anything about where the problem could be? JOGL? Xith? Did you notice the delay just on ATI gfx cards (like I did) ?