A library is more or less a set of tools meant to help you make games quicker. For example
- reading XML files
- getting access to OpenGL (LWJGL, JOGL)
- handling game objects
- handling game menus and UI
- a full game engine
With a game engine you don't have to code so much since the engine handles most of it. You pretty much only have to code things like game logic. The real work is filling the engine with content (images, menus, textures, 3D models, game logic, game content (items, skills, etc)). A game engine often comes with things like level editors and other editors for creating resources usable by the engine. Usually, the easier an engine is to use the more limitations there are. For example, a 2D game engine can't handle 3D graphics but is obviously simpler to use. Sometimes an engine allows you to work outside it or extend it, for example like how it's possible to do normal OpenGL calls while using Slick2D to do any kind of rendering (3D or 2D) you want.
Making your own complete game engine is a pretty bad idea in my opinion. It's a lot more work than most people think, especially the surrounding tools and deciding which features to implement, and in the end you might realize that it's not exactly what you wanted. You'll also not end up with a complete game, just an engine so the "reward" for your work is pretty much just the ability to make a game (hopefully) quicker than before. The chance that someone else is going to base their game on your engine is also pretty much 0 since not many people are willing to bet on a new unproven game engine.
I'd instead encourage you to write much smaller "libraries" for personal use. In other words write reusable code. I have a voxel renderer which I could easily use/modify to render a Minecraft world file (if I knew how to read a Minecraft save file that is
). I have a threading library I can use to multithread my games. I have a few classes abstracting things like OpenGL textures, tiled images, shaders, 3D cameras, etc. I have a frustum culler that can be used to check if objects actually will end up on the screen. I have a few pathfinding algorithms implemented, a fog-of-war renderer and some GUI stuff using TWL.
Many things can be shared between games that seem to be very different. A 2D strategy game needs pathfinding for the units, but so does a 3D first person shooter for AI controlled players. The same pathfinding "library" can be used in them if they're written correctly or with just slight modifications. Building up a bag of tools will help you make games quicker, but of course don't just make tools all day.
Make them as you make games. Again, just write reusable code.
