Hence, on a lot of scene switching, on alot of the scenegraphs out there, with VBO's enabled...the memory goes up, the fps goes down because the driver may not change new VBO's for old ones and move the new ones in system ram hence on every render call, the gfx has to fetch the data through the system bus in the system ram. And the same problem applies to Texture id's too...
This is not what happens. The old ones is moved to system ram and the new ones is uploaded to the gfx card. The opengl driver does this behind the scenes. That is why some games uploads all the texture at startup. The driver manages what needs to be in system and gfxcard memory. All the game have to do is make sure all the textures will fit in system ram, and that it don't use more than the gfx card can hold from frame to frame.
Ofcourse, you will run out of system ram eventually if you have a memroy leak.
I can see a serious memory leak regarding VBOs and the GC. When a Geometry class gets isolated and hence GC'ed, and that geometry class is tied with VBO's (i.e. has a VBO id), the required calls to free up the VBO and delete the buffer from VRAM/wherever else wont be called and the VBO id will be lost for ever
It is the programmers job to make sure there is no memory leak. It is a bug on your part if you loos all references to an object containing the texture/vbo id. One solution would be to delete the id when the object is removed from the graph. I'm sure it's more complicated than that, but it has to be dealt with. Using the finalizer is one lazy way of dealing with it. It will work but there are a couple of traps. First it may take a long time before it is called depending on the gc. Also your relying on the java heap gc to delete system resources wich is not good. Finally you can not call opengl from the finalizer thread. The finalizer have to tell the gl thread to delete.
In the end you just have to make sure you clean up your own mess.