Edit: What I'm noticing is that when the X rotation of the camera changes, then rotate the Y, that's when the angle of the camera starts to get weird. Is there a way to force the Z of the camera to stay up-right?
Edit2: I'm seeing what's going on with the camera, basically when I go to the object, rotate its yaw then zoom out as my code does the angle the angle stays locked at the same location, I need to update the yaw and angle relative to where the camera is, how is that done? A better way to put it is that when the camera moves out the angle does not update with the camera's new translation.
Edit3: I got it to work, what was happening was that the other method was not setting the rotation matrix to its current angle/yaw so it was rotating on its original axises, calling the standard rotateX/rotateY gets that to work just fine along with declaring lookAt ever update as well.
//Get position between camera and player