That's definitely better than my implementation. If I move the mouse perfectly side to side it looks nice along with straight up and down, however if I move the mouse sideways it starts to flip and turn in very odd directions.
Edit: What I'm noticing is that when the X rotation of the camera changes, then rotate the Y, that's when the angle of the camera starts to get weird. Is there a way to force the Z of the camera to stay up-right?
Edit2: I'm seeing what's going on with the camera, basically when I go to the object, rotate its yaw then zoom out as my code does the angle the angle stays locked at the same location, I need to update the yaw and angle relative to where the camera is, how is that done? A better way to put it is that when the camera moves out the angle does not update with the camera's new translation.
Edit3: I got it to work, what was happening was that the other method was not setting the rotation matrix to its current angle/yaw so it was rotating on its original axises, calling the standard rotateX/rotateY gets that to work just fine along with declaring lookAt ever update as well.
float distance = lock.distance(camera.getPosition());
float yRotation = -(dx * .005f);
float xRotation = -(dy * .005f);