Yes I'm not interested in ways to detect if there is a collision. I've already dealt with this.
For the beginning I don't want to make it too complicated, because I'm just started to learn OpenGL and need a basic system for that.
Roll back the time seems to be the thing that namrog proposed. I think it's the easiest way.
To push the two objects apart I need to keep track of the movement vectors, right? Also I think this would be only useful for entity - entity collision as walls aren't supposed to move

I'm not scared of math as long as it's not too hard

Your link seems to be very helpful and I will definitely safe it for later, but for the moment this seems to be a bit too much as I only try to have a basic system to handle collision so that I can move around properly. You still have a good point that one collision handling may cause another one. I'll keep that in mind. Thank you very much!