I'd also suggest that using a copying sweep would be the fastest method still. That is, as you iterate the arraylist, copy the elements into a new arraylist that are still "live", and then switch to that arraylist, like a double-buffer. This should make absolutely the best use of caches.
Cas

That's an awesome idea. Even with 100,000 elements, live at once, your only going to be using around 390kb of memory.