diff --git a/docs/source/_static/tut5_mesh-warper-organic-screenshot.png b/docs/source/_static/tut5_mesh-warper-organic-screenshot.png new file mode 100644 index 0000000000..e409a11ce7 Binary files /dev/null and b/docs/source/_static/tut5_mesh-warper-organic-screenshot.png differ diff --git a/docs/source/_static/tut5_mesh-warper-screenshot.png b/docs/source/_static/tut5_mesh-warper-screenshot.png new file mode 100644 index 0000000000..d02da6e953 Binary files /dev/null and b/docs/source/_static/tut5_mesh-warper-screenshot.png differ diff --git a/docs/source/_static/tut5_open-mesh-warper-from-splash-screen.png b/docs/source/_static/tut5_open-mesh-warper-from-splash-screen.png new file mode 100644 index 0000000000..50e7b343d0 Binary files /dev/null and b/docs/source/_static/tut5_open-mesh-warper-from-splash-screen.png differ diff --git a/docs/source/conf.py b/docs/source/conf.py index 2055075060..613f501fe1 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -38,6 +38,9 @@ # This pattern also affects html_static_path and html_extra_path. exclude_patterns = [] +# enable numbering each figure (e.g. "figure 1", "figure 2" in caption) +numfig = True + # -- Options for HTML output ------------------------------------------------- diff --git a/docs/source/index.rst b/docs/source/index.rst index 5d2b512f7e..a31d7966ab 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -47,6 +47,7 @@ Table of Contents tut3 tut4 tut5 + tut6 .. toctree:: :maxdepth: 2 diff --git a/docs/source/tut5.rst b/docs/source/tut5.rst index 5bdca9d0ae..a597f055d8 100644 --- a/docs/source/tut5.rst +++ b/docs/source/tut5.rst @@ -7,8 +7,20 @@ Use the Mesh Warper In this tutorial, we will be using the Mesh Warping UI to perform landmark-driven mesh warping using the Thin-Plate Spline algorithm (`explanation `_, `literature source `_). The UI -provides tooling for loading two meshes and creating landmark pairs between them. +provides tooling for loading two meshes and creating landmark pairs between them: +.. _cylinder-warp-example: +.. figure:: _static/tut5_mesh-warper-screenshot.png + :width: 60% + + The mesh warping UI, which shows the source ("reference", left) destination ("target", middle) and result ("warped", right) meshes. + Here, the warping quality is low. This is because of the low triangle and landmark count. + +.. figure:: _static/tut5_mesh-warper-organic-screenshot.png + :width: 60% + + Same as :numref:`cylinder-warp-example`, but showing an example of warping a clavicle bone. This example has many paired (left-to-middle) landmarks in a + variety of locations along the surface of the bone, which improves the warp quality (right). Prerequisites ------------- @@ -23,61 +35,124 @@ Prerequisites Topics Covered by this Tutorial ------------------------------- -- A general overview of the Thin-Plate Spline technique -- A general overview of the mesh warping UI -- Using the mesh warping UI on a basic anatomical example +- An overview of the Thin-Plate Spline technique +- How OpenSim Creator's mesh warping UI works with that technique +- A concrete walkthrough of using the UI on an anatomical mesh -The Thin-Plate Spline Technique -------------------------------- +The Thin-Plate Spline (TPS) Technique +------------------------------------- -TODO: Basic overview of the high-level idea (minimal warping of a thin plate of metal), how -it relates to the maths (fitting points in n-dimensions), how it relates to landmarks (3D corresponding points) and -where meshes (they're clouds of points that the resulting warp polynomial can be applied to) and "non-participating landmarks" (e.g. one-off datapoints) -fit into the picture. +.. note:: + This section isn't going to explain the Thin-Plate Spline (TPS) technique in + extensive detail. Instead, it will provide a simplified explanation + that's good enough to get a rough idea of what's happening when you use the + mesh warping UI. -The Mesh Warping UI -------------------- + If you want something more, we recommend consulting the `Relevant References`_, + which lists a variety of content related to the TPS technique. -TODO -Opening the UI -~~~~~~~~~~~~~~ +Imagine placing a thin plate with points along its surface onto a table. Now +imagine that each surface point has a corresponding "target" point somewhere +in 3D space. You could minimize the distance between those corresponding points +by physically bending the plate, but you'll need to figure out the optimal way +to bend it. -TODO +The TPS technique models this principle by: -Panels in the UI -~~~~~~~~~~~~~~~~ +- Describing "bending" the plate as a bounded linear combination of some + basis function, :math:`U(v)`. The `original paper `_ + used :math:`U(v) = |v|^2 \log{|v|^2}`, but `other sources `_ use :math:`U(v) = |v|`. +- Treating the problem of transforming "source/reference" points (landmarks), + :math:`x_i`, to "destination/reference" points (landmarks), :math:`y_i`, as an + interpolation problem. +- Solving the coefficients of that linear combination while minimizing the + "bending energy". `Wikipedia example `_: -TODO +.. math:: -Points of Interest in the UI (import, export, blending, etc.) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + E_{\mathrm{tps}}(U) = \sum_{i=1}^K \|y_i - U(x_i) \|^2 -TODO +The coefficients that drop out of this process can then be used to warp any +point in the same space. **However**, remember that this explanation is +simplifying things, and that you should read `Relevant References`_ if +you want to learn more. +Here's how concepts from TPS apply to OpenSim Creator's mesh warping UI: -Example: Warping a Femur +- OpenSim Creator's mesh warper uses the TPS technique. It's conceptually + handy to think about how that can be used to "bend" mesh surfaces. + +- **Source Mesh** and **Source Landmark** refer to data in the "reference", or + "source" space. Each source landmark requires a corresponding destination + landmark with the same name. + +- **Destination Mesh** and **Destination Landmark** refer to data in the "target", or + "destination" space. Each destination landmark must have a corresponding + source landmark with the same name. + +- **Warp Transform** is the product of the TPS technique after pairing the + landmarks and solving the relevant coefficients. The transform can be applied + to point data to warp it. E.g. in the mesh warping UI, the transform is applied + to the source mesh to produce the result mesh. It's also applied to "non-participating landmarks" + to produce warped points. + +- **Result Mesh** is the result of applying the warp transform to the source mesh. + +- **Non-Participating Landmark** is a landmark in the source mesh's space that + should be warped by the warp transform but shouldn't participate in solving + the TPS coefficients. + + +Opening the Mesh Warping UI +--------------------------- + +.. figure:: _static/tut5_open-mesh-warper-from-splash-screen.png + :width: 60% + + The mesh warping UI can be opened from the main splash screen of OpenSim Creator + (highlighted red). + + +Mesh Warping UI Overview ------------------------ -TODO +.. figure:: _static/tut5_mesh-warper-organic-screenshot.png + :width: 60% + + Screenshot of the mesh warping UI, with the (optional) "Landmark Navigator" window + open on the right-hand side. -Load Example Meshes -~~~~~~~~~~~~~~~~~~~ +Although the TPS technique (explained above) only actually requires paired +landmarks in 3D space, the mesh warping UI pairs those landmarks with the surfaces +of meshes. This pairing is natural when working with physiological meshes, because +"a pair of 3D points" isn't as useful as "the most posterior point on a femur's +lateral condyl in two CT scans" when we want to use those correspondences to +warp associated data (e.g. muscle points). -TODO +The mesh warping UI separates the information into three panels: -Step-By-Step Explanation -~~~~~~~~~~~~~~~~~~~~~~~~ +- **Source**: "source", or "reference" meshes and landmarks. These are where + you're starting from. In this panel, you can ``Import`` meshes, landmarks, + and non-participating landmarks (datapoints, such as markers, that should + be warped, but shouldn't participate in fitting TPS parameters). -TODO -Warping Single Points (markers, muscle points) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +* TODO: Explain top-level UI design (what each panel does) +* TODO: Explain key features (import, export, blending, etc.) + + +Example: Warping a Femur +------------------------ + +* TODO: Provide + Load Example Meshes +* TODO: Step-by-step explanation +* TODO: warping single points/non-participating landmarks -TODO +.. _Relevant References: Relevant References ------------------- @@ -133,3 +208,4 @@ the algorithm. .. _3D Thin Plate Spline Warping Function: https://uk.mathworks.com/matlabcentral/fileexchange/37576-3d-thin-plate-spline-warping-function .. _3D Point set warping by thin-plate/rbf function: https://uk.mathworks.com/matlabcentral/fileexchange/53867-3d-point-set-warping-by-thin-plate-rbf-function .. _A Practical Guide to Sliding and Surface Semilandmarks in Morphometric Analyses: https://doi.org/10.1093/iob/obz016 +.. _SemilandmarksInThreeDimensions: https://doi.org/10.1007/0-387-27614-9_3 diff --git a/docs/source/tut6.rst b/docs/source/tut6.rst new file mode 100644 index 0000000000..6413739d20 --- /dev/null +++ b/docs/source/tut6.rst @@ -0,0 +1,50 @@ +.. _tut6: + + +Use the Model Warper +==================== + +.. warning:: + + **Model Warping is 🪄 experimental 🪄.** + + You can try it out, get a feel for it, but beware that some things (e.g. + the ``.warpconfig.toml`` file) aren't established yet, which means they + may change, as we rollout more robust designs. + + +TODO: overview and screenshot, top-level motivations, etc. + +Prerequisites +------------- + +TODO: model editing, mesh warping, a basic understanding of Thin-Plate Spline + + +Topics Covered by this Tutorial +------------------------------- + +* TODO: how to ensure a model is warp-able +* TODO: basic example of a two-body model +* TODO: more advanced example walkthrough +* TODO: limitations, references, future work + + +Warpable Model Requirements +--------------------------- + +TODO: explain what the model warper can/can't warp. Explain ``StationDefinedFrame`` +and limitations around warping frames, muscle scaling, etc. + +Basic Example: Two-body model +----------------------------- + +TODO: provide a very stripped-down model that meets the requirements for warp-ability + + +Advanced Example: Many-Bodied Model with Custom Requirements +------------------------------------------------------------ + +TODO: an example model that requires the user to specialize/specify customization +in the warp config (e.g. tell the warp engine to skip some steps, warp X using +technique Y, etc.)