Questions related to a mesh object's position (how to getPosition, getRotation) and selecting them with a trackpad or touchscreen

[SugarRayLua]

Great to see how far the rgl package has come since I last used it and now with the additional features from the rgl2gltf package!

To use R and rgl more though as a 3D graphics engine, I'm trying to figure out how to do the following related to mesh object(s):

1. How does one get a mesh object's position (e.g. center of mass) in the x,y,z coordinate space? I understand one can do:

rgl.attrib(meshObject, "vertices")

to retrieve their position, but I'd like to be able to keep track of the meshObject as a whole, especially if I want to move an object around using:

translate3d(meshObject, x, y, z)

2. Similar to the above question, is there a way to get a mesh object's rotation (especially if I want to keep track of how rotated the meshObject is)?

3. Is there a way to use either a trackpad or a touchscreen to accomplish something similar to the select3d() function?

I'd like to be able to use such a function in a loop to to detect user selecting a meshObject in the scene (i.e. to detect that a user touched on windowSpace that corresponds to volume enclosed inside of a meshObject)

Thanks!

Have a good week :-)

[dmurdoch]

1. You use rgl.attrib() to get information about objects that have been drawn. It really has nothing to do with mesh objects, which are just R objects with class inheriting from mesh3d. So to find a measure of the center of a mesh object, I'd do something like this:

   m <- cube3d()
   vertices <- asEuclidean(m$vb)
   centroid <- rowMeans(vertices)

See ?mesh3d for a brief discussion of the format of a mesh object, and the rgl vignette for more.

2. Mesh objects don't record their rotation. If you want that, you'll need to keep your own record of what rotations you've applied.
3. I don't know!

[SugarRayLua]

Thank you very much. I'll try your suggestions. :-)

[SugarRayLua]

@dmurdoch, I've done more reading on matrix transformations: at your convenience would you clarify if I understand the difference between the rgl package capability and a 3D engine such as Unity or Unreal that:

Unity and Unreal engines allow the programmer to read the values of the transformed matrices that result from applying transformation to meshes but rgl does not (and that is why in Unity or Unreal the programmer can use functions to "read" what the positions, scale or rotation are of their mesh objects in world space but the programmer cannot do that with rgl)? Or am I misunderstanding the concept described in this post:

https://gamedev.stackexchange.com/questions/189896/how-mesh-transformation-works-under-the-hood

Or, does rgl.attrib(3dmesh, "vertices") actually retrieve transformed positions of vertices if I done some transformation matrix commands on them? If so, then I can understand your previous explanation how it just might take a little longer (than if had one command such as in Unity to obtain a mesh's worldspace position) to calculate the centroid of the transformed vertices revealed in rgl.attrib function?

I don't mean to request a lengthy explanation and understand if too advanced for a novice 3D programmer like myself but was hoping I might be on the right track in understanding what I'd need to do if I wanted to try and do some more advanced 3D programming with rgl (for instance, creating scenes with multiple moving objects and writing code to store the locations of those objects, writing simple physics functions for how those moving objects might interact, etc.). I understand such ideas were not likely the goal of the 3d functionality of the rgl package but was curious as I suspect some other users of the package might be on far one could go with rgl and 3D programming with R.

Thank you.

[dmurdoch]

I'm not familiar with Unity or Unreal.

In rgl, there are two ways to rotate or otherwise transform a mesh. You can apply a transformation to the mesh3d() object before plotting it. (You use shade3d() or plot3d() or some other fns to do the plotting). After you've plotted it, the coordinates don't change.

The other way to transform it is to apply the transformation to the scene or subscene where the object was plotted. Then it appears to move, but rgl.attrib(id, "vertices") will still return the coordinates that you plotted. You can retrieve information about the transformation that was applied using par3d(). See the section Rendering in the ?par3d help page for the fairly complicated way those transformations are applied.

You say that those engines allow you to retrieve the transformation of an object. That's not possible in rgl, because the first transformation that was applied to the mesh3d() data isn't recorded at all, and the transformations retrieved by par3d() apply to the whole scene, in which an object can appear several times with different transformations (e.g. if producing a stereo view, it needs to have two different projection matrices). Typically an object in a scene wouldn't appear more than once with different modelMatrix values, but rgl would let you do that if you wanted to.

If you want to duplicate the programming model that you describe, then the idea would be to put each collection of rgl objects that form a single solid real-world object into a separate subscene, and manipulate the matrices associated with that subscene to move the objects. That's how rgl2gltf handles complicated objects in example(playgltf). That package doesn't really have user-level support for authoring GLTF models. You'll have to write that yourself. If you need changes to rgl2gltf to support what you're doing, propose them as issues there.

[SugarRayLua]

Thank you very much, @dmurdoch-- I think I'm beginning to understand rgl better and how I might track/find what I'm looking for (mesh positions) and meanwhile learning more about matrix transformations for Open GL. I ended up following what you suggested with only a slight modification and will relate back to 3D game engine terminology that I'm more used to:

It seems the two different ways of transforming a mesh type object in rgl are:

1. transform the object in its "local space" coordinate system before plotting the mesh object:

mesh <- translate3d(cube3d(color = "green", tag = "greenBox"), 0, 0, -2)

will translate the green cube mesh 2 units down the z axis (into the screen) with the result that when render in rgl

shade3d(mesh)

the green cube mesh will be position 2 units down the z axis in the "world space" rgl scene

2. transform the mesh object's position in the "world space" rgl scene by setting up a "userMatrix" in the rgl scene to do such a transformation:

par3d(userMatrix <- translationMatrix(0, 0, -2))

will then translate the green mesh 2 units down the z axis after having already been situation in the rgl "world space" scene

despite the end result looking the same on the screen, I can tell that the transformation process was different by:

3. using the matrixSequence() function from the rgl2gltf package as follows:

matrixSequence("greenBox") which nicely shows me three matrices:

"Id": the x,y,z coordinates of the green cube mesh in its "local space" coordinates

"Matrix" which appears to be the product of all the transformation matrices there were applied to the green cube mesh's local space coordinates to generate the final green cube mesh's coordinates in the rgl "world space" scene. (It appears there is a default transformation matrix assigned to the rgl scene's "userMatrix" that looks like it helps position any objects rendered in an rgl scene for better viewing [appears to slightly rotate the object placed at the 0,0,0 center of the rgl scene "world space")

"Transformed" the resultant x,y,z coordinates of the green cube mesh's vertices in the rgl "world space" scene after the "userMatrix" transformations noted above have been applied to the vertices positions.

I mention all of the above because one of my original questions was whether or not a programmer could determine the location of a(each) mesh object in the rgl scene world space. Based on the above, I believe a programmer could determine the location of any mesh object in the rgl scene by doing:

4. matrixSequence("tagOfMeshObjectInterestedIn") and then looking at the the value in either the 1st, 2nd or 3rd row of 4th column of the matrix labelled "Matrix" which should be the x, y, and z positions, respectively, of that mesh object in rgl "world space".

Progammatically it looks like one could also retrieve those values but it looks a bit more complicated as it seems those values would be stored as follows:

matrixSequence("tagOfMeshObjectInterestedIn")[[1]]$userMatrix[["7"]][#,4] where # = 1,2, or 3 corresponding to the x,y,z position coordinates of the mesh object in rgl "world space".

(Obtaining the mesh object's rotation could also likely be obtained from different cells of that userMatrix mentioned above but would likely be more complicated as one would need to convert the values from radians back to degrees and noting that the values stored in those cells are either sin or cos of the different axis rotations)

I found that if I get lost tracking multiple transformations that I've applied to a mesh object, if I saved the default userMatrix transformation that I mentioned earlier by executing:

5. defaultTransform = par3d("userMatrix") then I can always reset my mesh's transformation by:
   par3d(userMatrix = defaultTransform)

I also realize as you mentioned that if I wanted to track the location (e.g. movement) of multiple 3D meshes in an rgl "world scene", since one can only apply world space transformation on mesh objects by using the par3d(userMatrix...) function which only works on a scene and not a specific mesh object per se, it would be best to set up each mesh separate in a rgl subscene and then specify each subscene when using par3d() to track each specific mesh object's position.

With the insight you've provided and what I've learned so far, I'll keep exploring the 3D "engine" capabilities of rgl. Hopefully the information contained in this issue might be helpful to other rgl users with similar interests :-)

[dmurdoch]

It sounds as though you've got the ideas now. The only part that might be incorrect is point 4: the entries in a 4x4 transformation matrix can be hard to interpret when translations and rotations are combined, and really hard if you allow any other kinds of transformations. I think what you say applies to the case where there are translations and no other changes.

And be careful about using <- in a function call as in point 2: that does an assignment in the global workspace, and uses the result as the first argument to the function, ignoring the name userMatrix. You use = to associate values with arguments.

[SugarRayLua]

Thanks, @dmurdoch.
I realized also that using the userMatrix to obtain the translations will also only be accurate if the programmer uses the second method you mentioned by doing all of their translations in the world space; if one instead were to translate the mesh in its own local space first before plotting/rendering then the transforming userMatrix would not contain translation information in its last column and thus not be helpful in obtaining a mesh's position.

Perhaps then the most accurate method to determine a mesh's current position in world space is what you initially mentioned by determining the centroid of the mesh by its current vertices and such vertices are listed in the "Transformed" vertices displayed using matrixSequence(meshTag) listed under my previous 3rd point.

Overall, I'll conclude that while it may be possible to determine at least a mesh's world space position in an rgl scene, it is likely much more complex to do so than a programmer would be used to doing so who is used to working with dedicated 3D engines like Unity or Unreal.

Have a good weekend :-)

[SugarRayLua]

P.S. @dmurdoch, I think I also found two clarification points that might be helpful to point out in the documentation for other users:

1. I figured out that to correctly rotate or translate mesh objects in "world space" using rotationMatrix and translationMatrix, one needs to actually use the transposed transformational matrices as follows:

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))
par3d(userMatrix = t(rotationMatrix(1.22, 1, 0, 0))) [positions the cube at the appropriate 1.22 radians/-70 degrees that
rgl's default mesh object rendering sets up]
par3d(userMatrix = t(translationMatrix(0, 0, -2))) [appropriately positions the mesh object 2 units deeper into screen z axis]

However, the non-transposed versions of the above statements do not correctly set up their intended transformations

I read that one might need to transpose the matrices since open GL is a "column major" order for matrix multiplications but that the transformation matrices stored in R are actually stored as "row major" order matrices

If one then wanted to correctly position a mesh object at 2 units deep to the screen and view the mesh object rotated -70 degrees one would then combine the transformed matrices like so in the recommended scale then rotate then translate order:

par3d(userMatrix = t(translationMatrix(0,0,-2) %*% t(rotationMatrix(1.22, 1, 0, 0)))

2. In this excerpt on how the camera angle in rgl is set up from the "get started" section of the rgl web page:

Camera angle
By default when you open a new plot with open3d:

The x axis goes from left to right.
The y axis goes from near the camera to far away.
The z axis goes from down to up.

*Are the axis orientations instead supposed to say:

The x axis goes from left to right.
The y axis goes from down to up.
The z axis goes from near the camera to far away?

I presumed so if rgl follows open GL conventions.

Fyi,

Have a good rest of your weekend.

[dmurdoch]

Regarding the initial orientation: The docs are correct. Run this code and you'll see it:

open3d()
plot3d(x=c(0,1,0,0), y = c(0,0,1,0), z = c(0,0,0,1), size=0.01, col=c("white", "red", "green", "blue"))

That plots a white sphere at the origin, and the others one unit out on the axes: red is X, green is Y, and blue is Z. The reason for this choice is that rgl was designed to follow statistical graphics conventions (e.g. as used by the base graphics function persp()), not OpenGL conventions. Generally Z is taken to be a function of X and Y, and plots show the function value as height.

For the other points about the matrices, you should generally multiply userMatrix by the transformation you want. You should read the ?matrices help topic carefully. It does explain that sometimes you need to transpose the matrix.

All of these issues are confusing because of the various different conventions used in different contexts. We've made an effort to keep rgl internally consistent, but our default choices are not necessarily the ones you might find in other contexts, and in some cases consistency isn't possible. (Statistical data traditionally has (x, y, z) values as rows of a matrix or dataframe, while meshes are much more efficient if they store points as columns.) We've also made an effort to let you set your own defaults if you want different ones. See the ?open3d help topic for how to do that.

[SugarRayLua]

Thanks, again, @dmurdoch -- I think I almost got it!

So, by statistical conventions, the plot3d() function: + x is to the right, +y is into the plane of the screen, + z is up
(which I realize also follows open GL coordinate system, only rotated in space so that z now points up but important in terms of plotting 3d points as you illustrated with your example)

However, it seems that at least when not specifically "plotting" 3d points but when using transformational matrices to move mesh objects, rgl goes back to using open GL naming conventions to the axes with + x right, + y up, + z coming out from the screen as demonstrated by the following code:

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))
par3d(userMatrix = t(translationMatrix(0,2,0)) %*% t(rotationMatrix(1.22, 1,0,0))) # moves cube up y 2 units

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))
par3d(userMatrix = t(translationMatrix(0,-2,0)) %*% t(rotationMatrix(1.22, 1,0,0))) # moves cube down y axis 2 units

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))
par3d(userMatrix = t(translationMatrix(2,0,0)) %*% t(rotationMatrix(1.22, 1,0,0))) # moves cube right x 2 units

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))
par3d(userMatrix = t(translationMatrix(-2,0,0)) %*% t(rotationMatrix(1.22, 1,0,0))) # moves cube left x 2 units

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))
par3d(userMatrix = t(translationMatrix(0,0,-2)) %*% t(rotationMatrix(1.22, 1,0,0))) # moves cube 2 units deep into screen z

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))
par3d(userMatrix = t(translationMatrix(0,0,1.5)) %*% t(rotationMatrix(1.22, 1,0,0))) # moves cube 1.5 units forward z from screen

Perhaps the above is another example of what you were talking about in terms of different contexts; the example you provided applied to plotting data on a 3D graph, the example I provided relates to the context of transforming mesh objects?

[dmurdoch]

In your case you are overriding the userMatrix value rather than modifying it. That's like starting from the identity matrix and modifying that. The coordinate with an identity matrix for userMatrix is the OpenGL one, and that's your starting point.

[SugarRayLua]

Thanks, again, @dmurdoch.

It sounds like the preferred method is to modify the userMatrix rather than overriding it. At your convenience, would you provide an example of how one would modify the userMatrix to translate a mesh object position rather than overriding it like I did? I couldn't seem to find an example of how to do that in the rgl examples. I think once I understand that, I'll have completed my understanding of the preferred method for how to get and modify mesh positions which will hopefully can be useful for others who are learning how to use the rgl package and can close the issue.

[dmurdoch]

Here are two examples. This one moves the cube (and everything else in the display) by 0.5 units to the right in the viewer's coordinates:

library(rgl)

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))

userMatrix <- par3d("userMatrix")
par3d(userMatrix = t(translationMatrix(0.5, 0, 0)) %*% userMatrix)

# restore it
par3d(userMatrix = userMatrix)

In these coordinates, X runs from left to right, Y goes up, and Z goes into the screen.

This second plot does it in the frame of reference of the cube, where the x-axis runs from the cyan side to the green side. It'll look similar, but if you manually rotate the cube just after plotting it, you'll see the difference:

open3d()

shade3d(cube3d(color = rainbow(6), meshColor = "faces"))

# Now manually rotate the cube

userMatrix <- par3d("userMatrix")
par3d(userMatrix = userMatrix %*% t(translationMatrix(0.5, 0, 0)) )

In the second display the y-axis direction is from blue to yellow (i.e. into the screen initially), and the z-axis is from red to magenta (i.e. up).

[SugarRayLua]

Interesting, that was helpful, @dmurdoch.

I was able to tell the difference better if I changed the y parameter in the translation matrix to 0.5 examples above: the first example worked like I had been doing and moves the cube up; when used in your second example it moves the cube into the screen.

Unfortunately, I don't have any formal training in linear algebra so have been learning all matrix algebra online. From what I read, your two examples make sense that they are different since matrix multiplication is not communicative and the order one therefore multiples the matrices gives different results. I've been starting with the userMatrix for my transformations as you've illustrated but had chosen to specify the default user matrix manually to help me understand better what is going on in the transformations (i.e. userMatrix = par3d("userMatrix") = rotationMatrix(1.22, 0, 0, 0) it seems rgl default)

I was choosing to do the matrix multiplications in the order suggested by the open GL tutorials I was reading:
transformationMatrixToApply %*% 3DobjMatrixToModify (rather than vice-versa)

I also read that by open GL conventions, one should aim to transform 3D objects in the following matrix multiplication order:
translationMatrix %% rotationMatrix %% scaleMatrix

However, from your examples, and since you mentioned statistical 3D plotting convention indicates x to be horizontal axis, y to run in and out of the viewing plane and z to be vertical axis, for my future transformation with transformation matrices in rgl, it seems best to do transformational matrix multiplication in the following order:

3DobjMatrixToModify %*% transformationMatrixToApply

Looking more over the available functions in rgl, I think it might be easier to stick with using the following transformation functions instead of trying to set up lower level transformation matrix multiplications:

scale3d(obj, x, y, z)
rotate3d(obj, x, y, z)
translate3d(obj, x, y, z)

instead of using par(3d) then could also more easily select individual objects in a scene to transform (e.g. move across the screen). However, unlike transforming the userMatrix in the par3d() function, I believe I would need to pop3d("3dObjectToMove") in order to erase the rendered mesh as previously drawn and then shade3d() the newly transformed (e.g. translated/moved) mesh.

Thanks again for all of your time and effort to try and explain these aspects of the rgl package.

The only trouble I have with that more seemingly simple method is that I can't seem to get the translate3d() function to work (scale3d() and rotate3d() work as expected and follow the +x horiz , + y in/out plane, +z vertical axis conventions):

shade3d(translate3d(cube3d(color = rainbow(6), meshColor= "faces"), 1.5, 0, 0))
shade3d(translate3d(cube3d(color = rainbow(6), meshColor= "faces"), 0, 1.5, 0))
shade3d(translate3d(cube3d(color = rainbow(6), meshColor= "faces"), 0, 0, 1.5))

I don't see any translation of the cube with any of those above commands. Perhaps I'm misunderstanding something about how to use them?

[dmurdoch]

No, the problem is that rgl is trying to be helpful, and that's not being helpful here. rgl always tries to look at the middle of the group of objects, because it's really confusing to users when the objects are out of the field of view. There's no hint about where to look to find them. So as the docs say, it includes "A matrix to translate the centre of the bounding box to the origin.". That means if you translate everything to the right, rgl automatically undoes your work and moves things back to where the user can see them.

If you want to do experiments with this, one way is to set up a few extreme points that never move, and then move objects around within those limits. The bounding box won't change, so rgl won't change the way it's looking, and you can see the effects of what you're doing.

[SugarRayLua]

Got it, @dmurdoch, thanks!

I didn't see that note about the re-adjusting to center of the bounding box under the transforming commands but see that it is under the informational section of how rgl renders under the par3d() information. So, it seems that if the user wants full independence to translate mesh objects, it may be best to use the par3d(userMatrix = transformationMatrixUserCreates) method (or else note that translate3d() will try and re-center an object in the viewport). Perhaps in a future iteration of rgl you could consider making it an option to set that "center viewport to center of bounding box" to off (keeping default being TRUE to center)? The translate3d() commands seems otherwise much simpler to use for programming and similar to commands used in other 3D engines.

I think I now have answers to all of my questions on how to use rgl to manipulate 3D meshes and hope the lengthy discussion and examples may be useful for other users with similar questions.

Please feel free to close this issue unless you had additional suggestions or points on this issue you thought were important.

Have a good rest of your weekend and upcoming week.