you are letting your threads sleep, right? otherwise the GC have a lot of garbage to collect. it only does that when there is too much garbage and hence the pause. by letting your threads sleep a little every now and then the GC can do a little collecting but very often resulting in smoother execution.
I added Thread.yield() (since I didn't want to give up a precious, whole millisecond) and it smoothed out. It still makes barely perceivable jumps every once in awhile, but I only notice because I have a trained eye. To everyone else, it's silky smooth. Thanks, mill!