1. IMHO the first step would be to learn shaders and the programmable pipeline, probably using a utility library to help you along the way. After that, you can start developing your own engine (if you feel it's necessary) or you can continue to use something like LibGDX for convenience.
I cover a variety of GL topics here:
https://github.com/mattdesl/lwjgl-basics/wikiYou can also read up on other modern GL series, like Arcsynthesis. Note that most of them will be GL 3.3+, and therefore not work on Mac 10.6, nor OpenGL ES (WebGL/iOS/Android).
http://tomdalling.com/blog/category/modern-opengl/http://open.gl/http://opengl-tutorial.org/Understanding the basics of the pipeline (vertex attributes, shaders, etc) are essential to developing "modern" GL apps. If you don't care to learn modern GL, and are OK with using deprecated code, you can stick to old tutorials like NeHe. This includes things like glMatrixMode, glLoadIdentity and GLU.
2. Usually it makes sense to use a utility library like LWJGL's matrix classes, or those in LibGDX. If you really want to write your own vector/matrix utilities, of course, nothing is stopping you...
3. More important than instancing would be culling faces, assuming you are doing something a la Minecraft. As sproingie said, VBOs are a better place to start.