Mouse picking mesh vertices in 3d viewport

xyz

Hello. I'm new to Godot, although fairly experienced with c++/opengl .

I'm prototyping something that requires player to select individual vertices on a procedurally generated mesh. Is there an easy way to do it in Godot?

Calinou

Using RayCasts and procedural geometry generation (likely SurfaceTool in your case), it's likely possible to achieve this. You can determine the closest vertex position to the raycast destination by looping over the vertices and comparing them with the raycast's collision position.

Note that you will likely have to regenerate the mesh when the user finishes moving a vertex, unless you're using ImmediateGeometry. However, ImmediateGeometry becomes slow very quickly, even with just a few hundred vertices.

xyz

Right. I've already implemented mesh generation via SurfaceTool.

There is no vertex moving involved. Just need to "inspect" vertices on mouse over and possibly select them.

I looked at RayCast docs. A way to approach it would be to setup a collider with concave mesh shape and use built in ray picking. However, according to documentation, this only returns world space hit point on the mesh. I'd still need to iterate through all the mesh data and do some math to determine which triangle was hit. Unless there is a way for engine to report that triangle.

Since this needs to happen each frame (because of mouse over), I'm suspecting all that iteration could be too much for GDScript and should be done in C++.

cybereality

You can try an old-school picking technique with colored objects. Usually you would render a series of objects (like let us say cubes) each one with a unique color and then the mouse pick just looks in the color buffer pixel, reads the color, and then have some look up table that translates colors to object ids.

In your case, you may have another screen buffer that draws with GL points, each a different color (might be possible, not sure about this). The color would indicate the vertex number, and you can store the local space coordinates or other values in a Dictionary. At least I think that should work.

xyz

Hmm, both of the suggested approaches work well for picking triangles, but may not be quite adequate for vertices.

Since vertices have no area at all, picking them boils down to checking if mouse is in vicinity of their 2D projections.

Ray cast approach that Calinou suggested may result in some erratic misses when picking vertices that appear on the mesh contour. If user clicks near enough to a vertex but still outside the mesh - ray hit test would fail. In some cases (e.g. spikes) the area to click becomes frustratingly small.

With second approach; drawing vertices as differently colored circles and then reading a mouse pixel from buffer, handling non-convex meshes becomes a problem. Depth buffer can't really help us here so we manually need to determine which vertices are obscured and then not render them. This could be problematic performance-wise since we need to iterate through all vertices and discriminate based on their visibility. This is a lot of ray casting per frame and on top of that we have no other choice but to render in some sort of immediate mode.

So after some pondering I came up with this:

For each vertex, project it into screen space and check if mouse is near enough to be picked. If yes, see if it's visible by casting a ray from camera to that vertex. If ray doesn't hit the mesh before reaching it, the vertex is visible and we add it to the list of hit vertices. If we end up with more than one vertex, pick the one nearest to the camera.

It's still a lot of iteration per frame (depending on vertex count), but costly visibility checks are brought down to minimum. Doing it in C probably wouldn't cause much of a performance hit.

In the end it boils down to general picking problem, but engine can at least handle visibility checks.

cybereality

Actually, that gives me another idea (on the color picking method). If you drew the model with each triangle a different color, then use the standard pixel buffer read which should be pretty fast. And you can use the depth buffer, so you don't have to worry about occlusion. Then, you know the triangle and you can just check the 3 vertices to see which one is the closest (again, should be simple math).

xyz

Yeah, but then we've got the same problem as with ray casting; possible misses for contour vertices. Because user would expect a hit if they click anywhere near enough to vertex's 2d projection, regardless if click happened inside or outside of the mesh silhouette. But by this method, clicking outside of the silhouette wouldn't register a hit.