Pixelprime's picture shows quite well the idea behind mouse picking in a 3D world. One can define that the mouse cursor is lying on the near clipping plane of the view frustum and defines a ray in addition to the camera origin.
This ray/line is unlimited in each direction. So to only "pick" stuff we can actually can see (stuff in the view frustum) we restrict our further search to the line-segment of this ray which lies in the frustum. You see this line segment in the illustration in the post above.
Further on I will describe the two task which are needed.Calculating the line segment
Create two 4D vectors by using the mouse position as the XY part, the projected near and far clipping plane values (-1, 1) as Z and 1 as W.
Then multiply them with the inverse view-projection matrix, get the difference for the direction and either the camera position or the near-vector as the start.
Now a more indepth explanation, but first some minor stuff about vector matrix multiplication:
- A 4x4 matrix can be seen as another coordinate-system or space. You probably already heard about object-, world-, eye-space in OpenGL.
- To multiply a vector with a matrix can be seen as to project that vector to the space the matrix is representing
- To undo such a projection one can use the inverse of the previously used matrix
Two calculate the two points we need (blue/red dots in the picture) we will use the OpenGL projection-space(we get there with the Projection-Matrix, glOrtho ...) to our advantage. The projection space is defined, so that every point in the view-frustum is in the range [-1, 1] for all dimensions.
i.e. a point with XY == (-1, -1) will be rendered to the left bottom pixel of the canvas, (0,0) to the center ...
Perhaps now it is already clear for some how we can easily calculate the positions of the red and blue points.
Both points share the same X,Y coordinates in the projection space, because both are covered by the mouse cursor. To calculate these coordinates we can simply divide the mouse position by the canvas size and do a little multiply and add:
float relativeX = mouseX / width;
float projectedX = relativeX * 2 - 1;
The Z coordinate is even more simple to derive. We just take -1 for the red and 1 for the blue point, see projection-space definition.
Now as a final step, we will need the inverse of the view-projection matrix. This is because we have the coordinates for both points in the projection-space, but need them in world-coordinates for our actual picking calculations.
The view-projection matrix projects points from world to projection space, so multiplying our calculated two points by the inverse gives us our solution of this chapter. Doing the line intersection test
As said in the introduction we want to do some picking with the line-segment defined by the red and blue points. The first thing to do is to define what we actually try to achieve. What we want to do is to find all objects which intersect with the line-segment and then select the object which intersection point is the most near to the camera origin.
How to do these two steps can vary greatly for each application. It totally depends on the organisation of the objects you want to pick, but let us take a quick look at the most common case with just a simple list of objects to test.
Say you have some game entities, like units in a RTS, you want to pick. The first thing you need to do is to define the physical boundaries of the entity somehow. You can do this by using some simple shapes like a sphere, AABB, OOBB or a capsule. Further more you can use some hierarchy of such shapes(like hitboxes for a character in a FPS. sphere for the head and boxes for the limbs) or just test against a polygon mesh. Depending on what you use, you will need some function which can calculate intersection point(s) between a line/ray and such shapes. Just use google("line AABB intersection") or ask/search in this forum.
The only difference between a line intersection test and one for a line-segment is that you have to check if the calculated intersection point lies between the two points of the segment.
As a little bonus a little thinking incentive for all the block-world lovers: Like you can calculate the pixels of a line you draw on a 2D canvas very easily, you can do the same with a 3D line in a 3D picture like a voxel-block-world is.