Merge pull request #11014 from BastiaanOlij/openxr_render_models_doc

Add documentation about OpenXR render models

tutorials/xr/index.rst

@@ -33,6 +33,7 @@ Advanced topics
    openxr_composition_layers
    openxr_hand_tracking
    openxr_body_tracking
+   openxr_render_models
 
 Godot XR Tools
 --------------

tutorials/xr/openxr_render_models.rst

@@ -0,0 +1,275 @@
+.. _doc_openxr_render_models:
+
+OpenXR Render Models
+====================
+
+A cornerstone of OpenXR's API design is being as platform agnostic as possible.
+A great example of this is OpenXR's action map system where XR runtimes
+have to support core interaction profiles to fall back on,
+if no interaction profile exists for the hardware being used.
+This ensures that OpenXR applications keep functioning even when used on
+hardware that didn't exist when the application was released,
+or that the developers of the application did not have access too.
+
+A consequence of this is that the application developer doesn't know with any
+certainty what hardware is being used, as the XR runtime could be mimicking
+other hardware.
+The application developer thus can't show anything in relation to the actual
+hardware used, the most common use case being showing the controllers the user
+is currently holding.
+
+Showing the correct controller models and having these models
+correctly positioned is important to a proper sense of immersion.
+
+This is where OpenXR's `render models API <https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XR_EXT_render_models>`_ comes in.
+This API allows us to query the XR runtime for 3D assets that are correct
+for the physical hardware being used.
+The API also allows us to query the position of this hardware within the
+tracking volume and the correct positioning of subcomponents of this hardware.
+
+For instance, we can correctly position and animate the trigger or show buttons
+being pressed.
+
+For those runtimes that support the
+`controller data source for hand tracking <https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XR_EXT_hand_tracking_data_source>`_
+, we can also correctly position the user's fingers and hand according to the
+shape of the controller.
+Do note that this works in combination with the
+`hand joints motion range extension <https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XR_EXT_hand_joints_motion_range>`_
+to prevent clipping of the fingers.
+
+OpenXR Render models node
+-------------------------
+
+The :ref:`OpenXRRenderModelManager<class_OpenXRRenderModelManager>`
+node can be used to automate most of the render models functionality. 
+This node keeps track of the active render models currently made
+available by the XR runtime.
+
+It will create child nodes for each active render model resulting in
+that render model being displayed.
+
+This node must have an :ref:`XROrigin3D<class_XROrigin3D>` node as an
+ancestor.
+
+If ``tracker`` is set to ``Any`` our node will show all render models
+currently being tracked. In this scenario this node must be a direct
+child of our :ref:`XROrigin3D<class_XROrigin3D>` node.
+
+If ``tracker`` is set to ``None set`` our node will only show render
+models for which no tracker has been identified. In this scenario this
+node must also be a direct child of our
+:ref:`XROrigin3D<class_XROrigin3D>` node.
+
+If ``tracker`` is set to ``Left Hand`` or ``Right Hand`` our node will
+only show render models related to our left or right hand respectively.
+In this scenario, our node can be placed deeper in the scene tree.
+
+.. warning::
+
+    For most XR runtimes this means the render model represents a controller
+    that is actually being held by the user but this is not a guarantee.
+    Some XR runtimes will always set the tracker to either the left or right
+    hand even if the controller is not currently held but is being tracked.
+    You should always test this as this will lead to unwanted behavior.
+
+In this scenario we can also specify an action for a pose in the action map
+by setting the ``make_local_to_pose`` property to the pose action.
+Use this in combination with an :ref:`XRController3D<class_XRController3D>`
+node that is using the same pose and you can now add a layer that allows
+you to deviate from the tracked position of both your controller and the
+related render model (see example below).
+
+.. note::
+
+    Combining the above with hand tracking does introduce the problem
+    that hand tracking is completely independent from the action map
+    system. You will need to combine the hand tracking and controller
+    tracking poses to properly offset the render models.
+
+    This falls beyond the scope of this documentation.
+
+Render model manager example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can download `our render models demo <https://github.com/godotengine/godot-demo-projects/tree/master/xr/openxr_render_models>`_
+which implements the setup described below.
+
+.. image:: img/openxr_render_models_setup.webp
+
+In this setup we find an :ref:`OpenXRRenderModelManager<class_OpenXRRenderModelManager>`
+node directly underneath our :ref:`XROrigin3D<class_XROrigin3D>` node.
+On this node our ``target`` property is set to ``None set`` and will handle
+showing all render models that are currently not related to our left or
+right hand controllers.
+
+We then see the same setup for our left and right hand so we'll focus on
+just the left hand.
+
+We have an :ref:`XRController3D<class_XRController3D>` that will track the
+location of our hand.
+
+.. note::
+
+    We are using the ``grip`` pose in this example. The ``palm`` pose is
+    arguably more suitable and predictable however it is not supported
+    by all XR runtimes. See the hand tracking demo project for a
+    solution to switching between these poses based on what is supported.
+
+As a child of the node we have an :ref:`AnimatableBody3D<class_AnimatableBody3D>`
+node that follows the tracked location of the hand **but** will interact
+with physics objects to stop the player's hand from going through walls etc.
+This node has a collision shape that encapsulates the hand.
+
+.. note::
+
+    It is important to set the physics priority so that this logic runs 
+    after any physics logic that moves the XROrigin3D node or the hand
+    will lag a frame behind.
+
+The script below shows a basic implementation for this that you can build
+upon.
+
+.. code-block:: gdscript
+
+    class_name CollisionHands3D
+    extends AnimatableBody3D
+
+    func _ready():
+        # Make sure these are set correctly.
+        top_level = true
+        sync_to_physics = false
+        process_physics_priority = -90
+
+    func _physics_process(_delta):
+        # Follow our parent node around.
+        var dest_transform = get_parent().global_transform
+
+        # We just apply rotation for this example.
+        global_basis = dest_transform.basis
+
+        # Attempt to move to where our tracked hand is.
+        move_and_collide(dest_transform.origin - global_position)
+
+
+Finally we see another :ref:`OpenXRRenderModelManager<class_OpenXRRenderModelManager>`
+node, this one with ``target`` set to the appropriate hand and
+``make_local_to_pose`` set to the correct pose.
+This will ensure that the render models related to this hand are properly
+shown and offset if our collision handler has altered the location.
+
+.. raw:: html
+
+    <div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; height: auto;">
+        <iframe src="https://www.youtube-nocookie.com/embed/_gNOd7wQ62M" frameborder="0" allowfullscreen style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"></iframe>
+    </div>
+
+
+Render model node
+-----------------
+
+The :ref:`OpenXRRenderModel<class_OpenXRRenderModel>` node implements
+all the logic to display and position a given render model provided by
+the render models API.
+
+Instances of this node are added by the render model manager node we used up
+above but you can interact with these directly if you wish.
+
+Whenever Godot obtains information about a new render model an RID is
+created to reference that render model.
+
+By assigning that RID to the ``render_model`` property on this node,
+the node will start displaying the render model and manage both the
+transform that places the render model in the correct place and
+animates all the sub objects.
+
+The ``get_top_level_path`` function will return the top level path
+associated with this render model. This will point to either the 
+left or right hand. As the top level path can be set or cleared
+depending on whether the user picks up, or puts down, the controller
+you can connect to the ``render_model_top_level_path_changes`` signal
+and react to these changes.
+
+Depending on your setup of the
+:ref:`OpenXRRenderModelManager<class_OpenXRRenderModelManager>` nodes,
+render models will be removed or added as their top level path changes.
+
+Backend access
+--------------
+
+The nodes we've detailed out above handle all the display logic
+for us but it is possible to interact with the data that drives
+this directly and create your own implementation.
+
+For this you can access the
+:ref:`OpenXRRenderModelExtension<class_OpenXRRenderModelExtension>`
+singleton.
+
+This object also lets you query whether render models are
+supported and enabled on the device currently being used by
+calling the ``is_active`` function on this object.
+
+The built-in logic implements the
+`interaction render model API <https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XR_EXT_interaction_render_model>`_
+that lists all render models related to controllers and similar
+devices that are present in the action map.
+It will automatically create and remove render model entities
+that are exposed through this API.
+
+As other extensions become available these can be implemented
+in a GDExtension plugin. Such a plugin can call
+``render_model_create`` and ``render_model_destroy`` to
+create the object that will provide access to that render
+model through the core render models API.
+
+You should not destroy a render model outside of this logic. 
+
+You can connect to the ``render_model_added`` and 
+``render_model_removed`` signals to be informed when new render
+models are added or removed.
+
+The core methods for working with this API are listed
+below:
+
+.. list-table:: Render modele extension functions
+   :header-rows: 1
+
+   * - Function
+     - Description
+   * - render_model_get_all
+     - Provides an array of RIDs for all render models
+       that are being tracked.
+   * - render_model_new_scene_instance
+     - Provides a new scene that contains all meshes
+       needed to display the render model.
+   * - render_model_get_subaction_paths
+     - Provides a list of subaction paths from your
+       action map related to this render mode.
+   * - render_model_get_top_level_path
+     - Returns the top level path associated with this
+       render model (if any).
+       Use the ``render_model_top_level_path_changed``
+       signal to react to this changing.
+   * - render_model_get_confidence
+     - Returns the tracking confidence for the tracking
+       data for this render model.
+   * - render_model_get_root_transform
+     - Returns the root transform for this render model
+       within our current reference space. This can be
+       used to place the render model in space.
+   * - render_model_get_animatable_node_count
+     - Returns the number of nodes in our render model
+       scene that can be animated
+   * - render_model_get_animatable_node_name
+     - Returns the name of the node that we can animate.
+       Note that this node can be any number of levels
+       deep within the scene.
+   * - render_model_is_animatable_node_visible
+     - Returns true if this animatable node should be
+       visible
+   * - render_model_get_animatable_node_transform
+     - Returns the transform for this animatable node.
+       This is a local transform that can be directly
+       applied.
+
+