Ok, I figured it out!!!
I need to find the normal of the two vectors and if the normal points in towards me (i.e. z is negative), then I have the angle I want, if it points away from me (i.e. z is positive) then I need to take 360-angle!!
Here is the updated code:
private float getAngle(Point p1, Point p2, Point p3)
float angle = 0, cosAngle = 0;
Vector3f A = new Vector3f(p1.x - p2.x, p1.y - p2.y, 0);
Vector3f B = new Vector3f(p3.x - p2.x, p3.y - p2.y, 0);
cosAngle = Vector3f.dot(A, B) / (A.length() * B.length());
angle = (float)(Math.acos(cosAngle) * (180/Math.PI));
AxB = Vector3f.cross(A, B, null);
if(AxB.z > 0) angle = 360f - angle;
Here are some pictures of what it did to different shapes!
This is the original shape that I was having trouble with.
Then I wanted a complicated shape to make sure it wan't going to puke.
And I wanted to make sure that it did something sane on a simple shape.