I have done three years of a computing with games design course, i find that courses never seem to delve deeply into games design (ironically), however my studies have helped me to understand the concepts behind designing a game.
You asked when is a good time to start games development
Simple answer: whenever you want.
More complex answer: Whenever you have a grasp on OOP, design patterns, datastructures and algorithms. OOP is the core to java and OOP is very powerful in the construction of programs (preaching to a choir here), once you have an understanding of OOP you should have enough to begin some basic games development, as you will understand API's that bit more. Someone mentioned having a grasp on art, this is not neccessarily as important to a game developer, in my opinion i consider the art aspect the "slapping the makeup on the finished product".
I wanted to touch on data structures and algorithm's
No one has yet mentioned AI, so i thought i'd give it a mention, if your designing a game an understanding of data structures and algorithm's will help you towards creation of pathfinding and other forms of AI learning that will give your games some life. Graphics like OpenGL are great but i think that a dedicated games programmer will focus on making a good Games Engine backed up with AI system. The art can be focused on near the end of development (unless you really need to see things graphically).