Converting Screen Coordinates to World Coordinates

[shochu]

Does anyone know how to do this or know of a good tutorial?  I'm trying to implement a very basic version of neverwinter nights movement system.   It might be a little too tough for me at this point but I'd like to give it a shot.  THank you very much.  (I know I've been posting a lot recently sorry bout that)

[Jens]

Have you read the GSG tutorial about picking? I don't know anything about the neverwinter nights movement system, but learning about picking may be a good start.

[Yuri Vl. Gushchin]

I think picking will now be big help with converting screen coordinates to world coordinates (if we are speaking about picking implemented using GL_SELECT as it done in Xith3D for now). Picking passes screen coordinates to OpenGL and then renders scene in GL_SELECT mode to get objects that match specified pick area.

To convert screen coordinates to world coordinates it is necessary to unproject screen coordinates using current projection matrix. But this will not give you the point in world coords, but ray - when you click on the screen you are missing 3rd coordinate (Z), so infinite number of points in 3D space correspond to single point in 2D space.

BTW, if you want to pick an object and come closer to it, you definitely need picking - here Jens is 100% right.

I also have no idea about neverwinter nights movement system ("neverwinter" sounds like I heard it somewhere but can not recall).

Please describe task in more details - it will be useful.

Yuri

Games published by our own members! Check 'em out!

[shochu]

Neverwinter nights is a 3D third person view.  In 2d it would be very similar to an isometric view.  THe camera is relatively locked but you can move it just a little bit up and down.  The way movement works is you click on the screen, then you're character runs to that point (in the 3d world).  Probably the same principle that any 3d RTS game uses such as Warcraft III.  Character movement is done by point and click.

The two main parts of this are determining where the user clicked and where the character needs to move to.

I've been able to find lots of great stuff on pathfinding but determing where in the world coordinate system to move the character based on the mouse click is stumping me.

-------

I was thinking about it a little more.  Would it work if I converted the 2d mouse click coordinate into the ray that it represents then figure out where the ray intersects the terrain to get my z coordinate?  (In my coordinate system the the y axis is "up" and "down", with z going "in" and "out" and x being left to right.  My terrain is super simplistic and is just a flat plane for now.

To be honest though if thats what I need to do I'm not quite sure how to do it.

[Jens]

To convert screen coordinates to world coordinates it is necessary to unproject screen coordinates using current projection matrix. But this will not give you the point in world coords, but ray - when you click on the screen you are missing 3rd coordinate (Z), so infinite number of points in 3D space correspond to single point in 2D space.

Can you explain a bit more detailed how this can be done in Xith3D?

[Yuri Vl. Gushchin]

The technique should be quite similar to use of gluUnProject function, that maps window coordinates to object coordinates:

The gluUnProject function maps the specified window coordinates into object coordinates using modelMatrix, projMatrix, and viewport. The result is stored in objx, objy, and objz.

I don't remember if Java implementation of gluUnProject exists in Xith3D, but this can be easily implemented because of C source of GLU is available, and proting should be trivial.

The only question will be choosing Z window coordinate, which should be I think either 0 or 1 - it's necessary to check which one is more appropriate.

I also sure that there are examples of using gluUnProject for plain OpenGL programming, which will be useful in this case.

Yuri

[Yuri Vl. Gushchin]

Also following thread will be useful:

http://www.java-gaming.org/cgi-bin/JGNetForums/YaBB.cgi?board=jogl;action=display;num=1069089567

Thread "Mouse-to-World Coordinates and Pick Rays???" in JOGL forum.

Yuri

[Jens]

I'm still not fully understanding what to do and how it works (probably I misunderstood it completely). There are actually two possible tasks. The first is that I want to know where I clicked on a certain Shape3D (no matter if it's visible or not). The second is that I want to find out which Shape3D I clicked and where. Let's assume I want to do the first.

I have the (x,y) coordinates of a mouse click and a Shape3D. I want to compute where I just clicked on that Shape3D. A mouse position doesn't have a z coordinate, so I actually compute the intersection of a ray and a Shape3D object. I call this a picking ray. However gluUnProject returns (x,y,z) coordinates (which is obviously not a ray). Does it return the coordinates where the picking ray hits the near clipping plane or what exactly? Why do I need to test which z mouse coordinate is more appropriate (0 or 1)?

If I want to implement this without using Jogl I need the current view and the getLocalToVWorld()-Transform3D of my Shape3D. Is this correct?

(Sorry for the late reply. I don't have much time these days.)

[shochu]

Hi Jens,
 Here's what I think you need to do.  You actually need to call gluUnProject twice.  The first time you call it with a value of 0 for winz.  This will give you the intersection with the near plane.  THen, you call gluUnProject again with a value of 1 for winz.  This will give you the intersection with the far plane.   Then you just subtract the two values to get the "picking ray".  The origin of the picking ray will just be the location of the view.   Once you have the ray you need to check what objects in your scene it intersects.  I'm not quite sure how to do this part yet but if I figure anything out I'll be sure to post it.  Hope that helps a little.

[Jens]

Yes, this makes it a bit clearer.

z=0 => near plane
z=1 => far plane

Why didn't I find this written down somewhere?

Actually there are still two problems: The first is to compute the intersection of a ray and a Shape3D. The second is that I don't know how to access the modelview, viewport and projection. I thought CanvasPeerImpl.getGL() and CanvasPeerImpl.getGLU() can be used to get GL and GLU objects and I can call "gl.glGetDoublev( GL.GL_MODELVIEW_MATRIX, modelview );" (similar for the other variables), but this just returns zeros.

[Btw. I personally think such a common function should be implemented in Xith3D. I don't really like having to use Jogl, because I want to be able to exchange Jogl and LWJGL when the LWJGL port is ready.]

Games published by our own members! Check 'em out!

[shochu]

I started working on this and I'm having problems figuring out how to get the ModelViewMatrix, ProjectionMatrix and ViewPort.  Does anyone know how I can get those values?  THanks for the help.
I tried the following but keep getting a null pointer exception on the gl.glGetDoublev line

1   2   3   4   5   6   7

double [] model = new double[16];                     CanvasPeer canvasPeer = universe.getCanvasPeer();             GL gl;             gl = ((CanvasPeerImpl)canvasPeer).getGl();             gl.glGetDoublev(GL.GL_MODELVIEW_MATRIX, model);

Okay so I checked and the gl I'm getting back is null which would explain the problem in the glGetDoublev line.  

Of course now I'm stuck trying to figure out why I'm getting a null value returned for gl.

[Yuri Vl. Gushchin]

You can access valid GL only from within display(...) method of CanvasPeerImpl. But, anyway, I don't think this approach is good because of you are trying to use renderer-specific functionality.

You can deduce the projection/model-view matrices from the following fact:

Xith3D JOGL renderer uses following schema to set matrices before rendering:

1   2   3   4   5   6   7   8   9   10

gl.glMatrixMode(GL.GL_PROJECTION);         gl.glLoadIdentity();         view.getProjection().get(trans);         gl.glMultMatrixf(trans);         view.getTransposeTransform().get(trans);         gl.glMultMatrixf(trans);         gl.glMatrixMode(GL.GL_MODELVIEW);         gl.glLoadIdentity();

and then for every shape sets its Model-View transform using shape.getLocalToVworld().

Yuri

[DavidYazel]

try view.getProjectionTransform().

[shochu]

Thanks for the response Yuri. I just wanted to check to make sure I understand.

For the Projection Matrix:

  1) I would create an identity matrix A
  2) Multiplay matrix A by the result of view.getProjection() to get matrix B
  3) Multiply matrix B by the result of view.getTransposeTransform() to get matrix C
  4) Matrix C would be the projection matrix

For the modelview matrix
   I can take any object in the world and get the model view matrix by using shape.getLocaltoVworld

Is this understanding correct?  Thanks for all the help!


Just read your post David ... there is no view.getProjectionTransform() but there is a view.getProjection which returns a Tranform3D and I'm trying that out right now.  Thanks for the tip.
~shochu

[Yuri Vl. Gushchin]

I just wanted to check to make sure I understand

Looks like OK, but definitely you have to test this

Yuri

[shochu]

I've been working on porting the gluUnProject function from the mesa version of OpenGL into my xith program and I'm getting a little confused when it comes to the modelview matrix and using gluUnProject.


As taken from the openGL redbook gluUnProject maps the specified window (screen) coordinates into object coordinates.

From what I've read it seems that you use gluUnProject to convert screen coordinates obtained through mouse clicks into world coordinates.  

I assumed we could do the following

   Call gluUnProject twice to get the picking ray
   Once you have the picking ray check where the     picking ray intersects the objects in your scene.  (In this case I want to know where it intersects the terrain)  


However, if you are really getting object coordinates then how does this help you in determing if the picking ray intersects anything?  Do you have to use the model view matrix for the object you want to test intersection with?  

Thanks in advance for any clarification. ~shochu

[tom]

gluUnProject will convert the screen coordinates into the space given by the modelview matrix. So if the modelview matrix contains only the camera transformation, you will get what most people would think of as "world" coordinates. If you call gluUnProject just before rendering a mesh, you will get the coordinate in the meshes object space.

A brute force way of checking if your ray hits a mesh is to check if the ray hits any of its triangles. A better way is to use a bsp-tree, but thats off topic When doing this test the ray and the mesh must be in the same space. If your ray is in world space, you would either have to transform the ray into object space, or transform the mesh coordinates into world space. The first option is obviously the fastest in most situations.

So it's a good idee to have the ray in object space. This means you have to create a ray for each modelview you use. So don't think of it as one ray in world space. But one ray for each object.

Hope this help.

[shochu]

I've got kind of a unique use so let me elaborate (Thanks for your reply by the way it helped a lot)

I'm making a 3D isometric game.  (I think that's an oxymoron) (Think Neverwinter Nights or any 3d RTS ala Warcraft 3)  The game is just for learning purposes of course.

For now the terrain is just a perfectly flat plane.
defined by Point(0,0,0) Normal (0,1,0)

When the user clicks on the screen I want to figure out where they are clicking on the terrain so I can decide where to move the avatar.

When the user clicks on the screen with the mouse I grab the screen coordiantes.  I then pass these to gluUnProject first with winZ set to 0 then winz set to 1.  

From this I can get the picking ray.

I'm a little confused as to what model view matrix I should use. The bottom right corner of the terrain is at point (0,0,0) and then grows from there.  Thus I thought that my world coordinates would coincide with the terrain coordinates.

But it seems that I need to use the Modelview matrix of the terrain?    If I use the modelview matrix of the terrain then is this giving me a ray which goes from the camera through the point clicked on by the user?  

If so then I was planning to then figure out where the ray intersected the plane.  When you say that if the modelview matrix only contains the camera transformation would this mean that it the same as the projection matrix? Thanks for your generous help! ~shochu

[DavidYazel]

There is another approach you can take.  Using the view transform and the perspective transform you should be able to translate the screen coordinates into world coordinates.  Then you could calculate a point which was behind the view but centered on it (eye point) and calculate a ray which went from there and passed through the point you first calculated. I *think* this would result in a ray you could use to calculate an interection with your terrain.  I really havn't thought deeply about this, but I think it would work.

[shochu]

Hi David,
  Isn't your suggestion pretty much the same as what I'm trying to do now?  I'm still having problems but will keep working on it.

On another note,
  I ported over the gluUnProject code for anyone that might want to use it.  I think it's pretty useful.  I'm pretty sure it works without any problems (the problems I'm having are related to how I use the gluUnProject ... i think)  I called it xithUnProject.

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53

/**        * Multiplies the 4x4 matrix 'M' by the column vector 'in.'  This is a helper function for the xithUnProject method        * @param M        * @param in        * @return out a column vector with 4 elements        */       private float[] transform_point(Matrix4f M, float[] in)       {          float [] out = new float[4];          out[0] =               M.getElement(0, 0) * in[0] + M.getElement(0, 1) * in[1] + M.getElement(0, 2) * in[2] + M.getElement(0, 3) * in[3];          out[1] =               M.getElement(1, 0) * in[0] + M.getElement(1, 1) * in[1] + M.getElement(1, 2) * in[2] + M.getElement(1, 3) * in[3];          out[2] =               M.getElement(2, 0) * in[0] + M.getElement(2, 1) * in[1] + M.getElement(2, 2) * in[2] + M.getElement(2, 3) * in[3];          out[3] =               M.getElement(3, 0) * in[0] + M.getElement(3, 1) * in[1] + M.getElement(3, 2) * in[2] + M.getElement(3, 3) * in[3];                    return out;       }             /**        * Converts the given screen coordinates into object coordiantes based on the given projection and modelview matrix.        * @param winX  screen coordinate x        * @param winY  screen coordinate y        * @param winZ  screen coordinate z        * @param model modelview matrix        * @param projection projection matrix        * @param viewPort viewport        * @param results puts the final object coordinates into the results array where results[0] = x, results[1] = y results[2] = z        * @return boolean stating whether the unprojection was succesful or not        */       public boolean xithUnProject(float winX, float winY, float winZ, Matrix4f model, Matrix4f projection, int []viewPort, float[] results){             float in[] = new float[4];             float out[] = new float[4];                         Matrix4f A = new Matrix4f();             in[0] = (winX - (float)viewPort[0]) * 2.0f / (float)viewPort[2] - 1.0f;             in[1] = (winY - (float)viewPort[1]) * 2.0f / (float)viewPort[3] - 1.0f;             in[2] = 2.0f * winZ - 1.0f;             in[3] = 1.0f;                         A.mul(projection, model);             A.invert();             out = transform_point(A, in);             if(out[3] == 0)                   return false;                         results[0] = out[0] / out[3];             results[1] = out[1] / out[3];             results[2] = out[2] / out[3];             return true;       }

Please let me know if you use it and think something isn't working properly.  

[tom]

But it seems that I need to use the Modelview matrix of the terrain?    If I use the modelview matrix of the terrain then is this giving me a ray which goes from the camera through the point clicked on by the user?  

If so then I was planning to then figure out where the ray intersected the plane.  When you say that if the modelview matrix only contains the camera transformation would this mean that it the same as the projection matrix? Thanks for your generous help! ~shochu

With the camera transformation I did not think of the projection matrix. The projection matrix transforms eye coordinates to a unit cube that is used for clipping.

I've not used Xith so I don't know how it works. But the modelview matrix that I'm refering to will transform coordinates in object space into eye (or camera) space.

This matrix includes the camera (View). It might be that you've not included the view matrix in your modelview sent in to the unProject function. But I'm only guessing. If so you have to multiply the view matrix with the matrix that transforms object coordinates into world coordinates.

Does this make sens?

[shochu]

Yup, that makes sense.  I hate keep pestering you with questions but here comes another one.  (A little more concrete this time)


I have an object on the screen located at (50,0,50) in world coordinates.  It is currently at the center of the screen.

I click on the object.

I then call gluUnProject using the projection matrix and using a modelview matrix which only contains the camera's transformation (because I want world coordinates back).  In addition I pass the in the viewport, the mouseX and mouseY coordinates and the value 1.0 for winz which I beleive is for the far clipping plane.

For the viewport I am using
viewPort[] = {0, canvasHeight, canvasWidth, canvasHeight}  The first two values should be the location of the lower left of the screen.  I know the default is 0,0 but I think in xith3d 0,0 is the top left so I adjusted it accordingly.

I get back from the gluUnProject call the following values.

targetx: 49.99645
targety: -40.57691
targetz: 15.948797

Then I call gluUnProject again with the exact same parameters except with winz of 0 for the near clipping plane and I get.

eyex:  50.00118
eyey:  -43.640434
eyez:  13.434351

I'm a little concerned because I thought that at least one of the values in target and eye would have to be the same since the only paramter that changed was the winz.  Is that assumption in fact correct?

[tom]

No It's not correct. Don't know exactly how to explain it. For one of the values to be the same, the ray would have to lie on one of the axis planes in 3d. Which is almost impossible to do even if you try

Btw. your eye and target looks correct. Does it work?

[kevglass]

Did you ever get this sorted out? I'd like to do a similar thing..

Kev

[kevglass]

Oh well, if anyone's interested, I found a way of getting it to work.. but its not pretty or super accurate

Kev

[shochu]

I'm interested! Can you post source?  Did you use gluUnproject or did you come up with another way to do it?

[kevglass]

Ok.. this is what I use (well roughly), I'd be happy if someone would tell me its wrong

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36

public Point3f toWorld(int x,int y) {         return toWorld(x,y,view.getFrontClipDistance());     }     public Point3f toWorld(int x,int y,float depth) {         float fov = view.getFieldOfView();         float width = canvas.getWidth();         float height = canvas.getHeight();                  float panelY = (float) (Math.tan(fov) * depth);         float panelX = panelY * (width/height);                  float xp = x / (width/2);         float yp = y / (height/2);                  Point3f pt = new Point3f((-1 + xp)*panelX,                                  (-yp + 1)*panelY,                                  -depth);                  Transform3D v = new Transform3D();         view.getTransform(v);                  v.transform(pt);         return pt;     }     public void castRay(int x,int y,float targetHeight) {         Point3f center = toWorld(x,y);         Point3f d = toWorld(x,y,view.getFrontClipDistance()+0.1f);         Vector3f forward = new Vector3f(center);         forward.sub(d);         // at this point the ray is defined by        // point "center" and vector "forward"     }

So, all I currently do is convert the screen coordinates to world coordinates based on the view transform and a given depth. Then based on a second depth. The vector these two points gives the direction of the ray...

It seems to work ok, there are some new screenshots at:

http://www.newdawnsoftware.com/martian

of what I'm trying to do.. which I think might match up with what you're doing.

Incidently, I also use this code to generate the geometry for the "rubber band" used to select a group of aliens

EDIT: Please excuse typos, a little christmas "spirit" has been injested at this point

Kev

[Ramses]

When I get more than one result from the View.pick() function, I'd like to be able to tell which object is "closest" to the screen. The easiest way I figured to do this is to convert the mouse click point to world coordinates and then compare distances between the converted world position and the node positions to find the shortest one.

But to be completely honest I didn't quite understand the discussion that took place in this thread   (i'm new to 3D).  Could shochu or anyone else please explain what the parameters in the "xithUnProject" function listed above are, and where I would get them from? (I understand what winX, winY and results are, but not the rest).

[gpothier]

To find which picked object is closest to the screen, you can use the PickRenderResult.getZMin and getZMax. They return the smaller and bigger distance between the eye and the object (they can be different if the object's geometry intersects the ray more than once).
At least, that's how I understand this.

[gpothier]

Here are two methods based on the previously posted xithUnmap: one that creates a ray in vworld coordinates based on a canvas (mouse) coordinate, and another one that convert a vworld ray to a given node's coordinate system.
It is working all right for me, but I cannot say I have tested it toroughly yet.
How about, however, including this set of method in a xith utility class?