I am only moving back objects if they collide solid objects.
That's alright but it is not necessarily always the case.
Someone wants to use your collision detection system for his spooky ghost hunting game. As it is part of the nature of ghosts, they can pass walls. So he wants no automatic back moving, but still be notified of wall collisions to play a nice graphic effect for instance. And for the hunters, fine, move them back when colliding with walls. But the decision can not be made by the collision check.
The game is slow for actual use itself. I can notice a half second delay in movement.
The most efficient way to find the cause is to use a profiler. VisualVM is even free. Studying source code before that is rather a waste of time.
Also how could I split it?
Just identify all various tasks like I did and create new classes for them.
collision detection system has no dependency on Java2d. It should be usable on its own without rendering stuff and the like and it should be usable in a multiplayer environment where players might hang out on different maps at the same time. Not possible when there is only one map with static attributes.
The way to load and store maps is another part that should be certainly extracted. There too many ways and flavors to do that.