And yes, 3D game programming is heavy math. if you don't know Vector Algebra its probably not something you want to try.
That is true if you want to do the next Doom 4 or Half-Life 3 or something. You can still do simpler 3D games that don't need the heavy math.
My math and vector algebra is not great (to say the least...), but I've still been able to develop 2 simple 3D games (Hyper Blazer and Cosmic Trip) and VirtuArcade.
If you want to do 3D, just dive into it and see how far you can get with your current algebra skills and learn more if required.