Since its hard to describe, so I drew it out:



Can someone help me code this pleease?

Or:

calculate the cross product m := a x b. This is pointing out of the plane you want your vector in. Now you need something to make it perpendicular to both m (so it is in your desired plane) and a. One such candidate is n := a x m.

In total, n = a x (a x b).

Ever heard of the "BAC-CAB" rule? It allows any double cross product (i.e. A x (B x C)) which would ordinarily involve the calculation of around the same complexity as two 3x3 matrices, to be expressed as simple inner products. In general, A x (BxC) = B(A.C) - C(A.B), where "." denotes the inner product (perhaps the term "dot" product is more familiar). Therefore,

n = a(a.b) - b(a.a).

Hehe, thanks. But really it's less general (and no more simple) than the Gram-Schmidt orthogonalization procedure which would also have been applicable. The cross products are cooler, though, because the method works even if one of the vectors is zero. Basically, the Gram-Schmidt method works by subtracting the projection of b onto a from b (which is not defined if a = 0):

n := b - [projection of b onto a] = b - a (a.b) /(a.a).

Then the previous result would follow directly from multiplying by (a.a). Normally, however, one would divide n by its length to obtain a unit vector. The Gram-Schmidt orthogonalization procedure consists of repeating this method for any number of vectors, so you can slowly form an orthonormal (i.e. all unit vectors and set and all perpendicular) set. Works in (vector) spaces of any (finite) dimension.

Sorry about all the math stuff, but no one ever talks about anything interesting in the physics forum.