My suggestion for reading material on transformation matrices: http://www.wildbunny.co.uk/blog/vector-maths-a-primer-for-games-programmers/
. And LWJGL has its own Vector and Matrix classes that should be sufficient for anything you need to do.
@opiop65 Rotation matrices aren't so hard if you think about them in the right way. I think of 3x3 matrices as defining a set of axes. The first row the x axis, the second the y and the third the z. So with the identity matrix:
1 0 0 X axis
0 1 0 Y axis
0 0 1 Z axis
Then a rotation (about a single axis) is a simple matter of trig. But the beauty of matrices is that a complex transformation is only as complex as its individual components. I won't bore/patronize you by talking through the maths but I feel like it's a good way to think about it. Just (what I think is) a handy hint.