|
@@ -3,5 +3,199 @@
|
|
|
The OpenXR hand tracking
|
|
|
========================
|
|
|
|
|
|
-This is just a placeholder
|
|
|
+Hand tracking is the process by which the position and orientation of the players hands are tracked,
|
|
|
+including the orientation of the players fingers.
|
|
|
+We can identify 3 categories of this:
|
|
|
+
|
|
|
+One, hand tracking through external sensors such as cameras.
|
|
|
+This is what Hololens, Quest, UltraLeap and similar devices do.
|
|
|
+This often results in very accurate tracking of all fingers of the players hands.
|
|
|
+
|
|
|
+Two, hand tracking through VR gloves.
|
|
|
+This method is still mostly experimental but is likely to gain popularity soon.
|
|
|
+Gloves often have good finger tracking capabilities but their real selling point is the ability to restrict movement.
|
|
|
+This allows the sensation of touch.
|
|
|
+Gloves are often also recognised as controllers and often will have additional controls such as buttons embedded.
|
|
|
+
|
|
|
+Three, inferred hand tracking.
|
|
|
+This has been the defacto approach since the early days of VR.
|
|
|
+As we know the player is holding a controller and we know the position of this controller,
|
|
|
+we can infer where to render the players hand.
|
|
|
+Fingers can be positioned based on the controls the player is interacting with.
|
|
|
+Many modern VR controllers have additional sensors to help determine finger positions on the controller.
|
|
|
+
|
|
|
+.. note::
|
|
|
+
|
|
|
+ Traditionally inferred hand tracking has been the responsibility of the game.
|
|
|
+ However the principles behind the action map have somewhat limited the viable options here.
|
|
|
+ Valve is currently the only XR Runtime that has implemented inferred hand tracking as part of the hand tracking extension.
|
|
|
+ There is an expectation that other XR Runtimes will follow this example in the near future.
|
|
|
+
|
|
|
+ Until then we recommend that if your game depends on inferred hand tracking,
|
|
|
+ to use the hand assets that are part of Godot XR Tools.
|
|
|
+
|
|
|
+Tracking through interaction profiles
|
|
|
+-------------------------------------
|
|
|
+
|
|
|
+Tracking the location and state of controllers are performed through interaction profiles.
|
|
|
+Bindings can be set within the :ref:`action map <doc_xr_action_map>`.
|
|
|
+
|
|
|
+However it is important to realise that in OpenXR controllers are bound to paths indicating the usage of these controllers.
|
|
|
+I.e. the controller held in the players left hand is bound to ``/user/hand/left``
|
|
|
+while the controller in the players right hand is bound to ``/user/hand/right``.
|
|
|
+And while not yet supported outside of the HTC tracker extension,
|
|
|
+it is likely OpenXR will be extended with paths such as ``/user/foot/left`` and ``/user/foot/right`` at some point.
|
|
|
+
|
|
|
+.. warning::
|
|
|
+
|
|
|
+ This paradigm therefore begs the question what happens to a controller that is not being held by a user.
|
|
|
+ There is no answer to this question yet, this is still being debated and the specification may change in the near future.
|
|
|
+ The behavior is thus undefined and can be different for different platforms.
|
|
|
+
|
|
|
+ The most common is that the controller will remain bound regardless of whether the player is actually holding the controller.
|
|
|
+
|
|
|
+ However there are runtimes, such as the Quest, that can unbind a controller when it is not being held by the user.
|
|
|
+
|
|
|
+ This may become the norm in the future and the expectation is that the action map system will be enhanced accordingly.
|
|
|
+
|
|
|
+The hand tracking extension
|
|
|
+---------------------------
|
|
|
+
|
|
|
+OpenXR has an extension that exposes hand tracking functionality.
|
|
|
+This extension allows a game to request the hand skeleton with all bone positions for each hand.
|
|
|
+
|
|
|
+.. figure:: img/openxr_hand_skeleton.webp
|
|
|
+ :align: center
|
|
|
+
|
|
|
+ Copyright (c) 2017-2022, The Khronos Group Inc. SPDX-License-Identifier: CC-BY-4.0
|
|
|
+
|
|
|
+The above image shows the joints that have to be provided by each XR runtime that implements this extension.
|
|
|
+
|
|
|
+Currently Godot exposes this functionality through the :ref:`OpenXRHand <class_openxrhand>` node.
|
|
|
+This is a helper node that will retrieve the hand tracking data from OpenXR and apply it to a skeleton in Godot.
|
|
|
+You select either the left or right hand through the ``hand`` property on this node.
|
|
|
+
|
|
|
+The hand asset itself has to be provided by the developer and is thus separate from the OpenXRHand node.
|
|
|
+You set the ``hand_skeleton`` on the OpenXRHand node to the skeleton it needs to apply the tracking data to.
|
|
|
+
|
|
|
+If supported by the XR runtime you can also set the ``motion_range`` property to limit how far the hand can close.
|
|
|
+
|
|
|
+The skeleton for this asset has to have the following bones:
|
|
|
+
|
|
|
+.. list-table:: OpenXR hand skeleton
|
|
|
+ :widths: 50 50
|
|
|
+ :header-rows: 1
|
|
|
+
|
|
|
+ * - Left hand
|
|
|
+ - Right hand
|
|
|
+ * - Palm_L
|
|
|
+ - Palm_R
|
|
|
+ * - Wrist_L
|
|
|
+ - Wrist_R
|
|
|
+ * - Thumb_Metacarpal_L
|
|
|
+ - Thumb_Metacarpal_R
|
|
|
+ * - Thumb_Proximal_L
|
|
|
+ - Thumb_Proximal_R
|
|
|
+ * - Thumb_Distal_L
|
|
|
+ - Thumb_Distal_R
|
|
|
+ * - Thumb_Tip_L
|
|
|
+ - Thumb_Tip_R
|
|
|
+ * - Index_Metacarpal_L
|
|
|
+ - Index_Metacarpal_R
|
|
|
+ * - Index_Proximal_L
|
|
|
+ - Index_Proximal_R
|
|
|
+ * - Index_Intermediate_L
|
|
|
+ - Index_Intermediate_R
|
|
|
+ * - Index_Distal_L
|
|
|
+ - Index_Distal_R
|
|
|
+ * - Index_Tip_L
|
|
|
+ - Index_Tip_R
|
|
|
+ * - Middle_Metacarpal_L
|
|
|
+ - Middle_Metacarpal_R
|
|
|
+ * - Middle_Proximal_L
|
|
|
+ - Middle_Proximal_R
|
|
|
+ * - Middle_Intermediate_L
|
|
|
+ - Middle_Intermediate_R
|
|
|
+ * - Middle_Distal_L
|
|
|
+ - Middle_Distal_R
|
|
|
+ * - Middle_Tip_L
|
|
|
+ - Middle_Tip_R
|
|
|
+ * - Ring_Metacarpal_L
|
|
|
+ - Ring_Metacarpal_R
|
|
|
+ * - Ring_Proximal_L
|
|
|
+ - Ring_Proximal_R
|
|
|
+ * - Ring_Intermediate_L
|
|
|
+ - Ring_Intermediate_R
|
|
|
+ * - Ring_Distal_L
|
|
|
+ - Ring_Distal_R
|
|
|
+ * - Ring_Tip_L
|
|
|
+ - Ring_Tip_R
|
|
|
+ * - Little_Metacarpal_L
|
|
|
+ - Little_Metacarpal_R
|
|
|
+ * - Little_Proximal_L
|
|
|
+ - Little_Proximal_R
|
|
|
+ * - Little_Intermediate_L
|
|
|
+ - Little_Intermediate_R
|
|
|
+ * - Little_Distal_L
|
|
|
+ - Little_Distal_R
|
|
|
+ * - Little_Tip_L
|
|
|
+ - Little_Tip_R
|
|
|
+
|
|
|
+.. warning::
|
|
|
+
|
|
|
+ The skeleton data returned from different XR runtimes are often not compatible which poses a problem for cross platform development.
|
|
|
+
|
|
|
+ For instance both Microsoft and Meta runtimes base the skeleton on the actual hands of the player.
|
|
|
+ These skeletons will thus conform to the size of the players hand.
|
|
|
+
|
|
|
+ Contrast that to Valve where a fixed skeleton is used if inferred hand tracking is applied.
|
|
|
+
|
|
|
+ We are still gathering experience on how best to deal with these differences in the platforms.
|
|
|
+
|
|
|
+When exporting to Meta Quest you need to enable the following setting:
|
|
|
+
|
|
|
+.. image:: img/xr_enable_handtracking.webp
|
|
|
+
|
|
|
+Body tracking
|
|
|
+-------------
|
|
|
+
|
|
|
+At the time of writing, OpenXR does not support body tracking as part of the core spec.
|
|
|
+We are expecting this to change in the near future as more and more body tracking solutions are hitting the market.
|
|
|
+
|
|
|
+For now the only option available here is through HTC trackers.
|
|
|
+There is an extension that becomes available if HTC trackers are supported by the XR runtime.
|
|
|
+These are fully exposed through the action map system.
|
|
|
+
|
|
|
+Godot has full support for these and you can setup the trackers in the action map.
|
|
|
+Each tracker is assigned a usage within the SteamVR interface.
|
|
|
+
|
|
|
+These are exposed through the following trackers:
|
|
|
+
|
|
|
+.. list-table:: HTC trackers
|
|
|
+ :widths: 100
|
|
|
+ :header-rows: 0
|
|
|
+
|
|
|
+ * - /user/vive_tracker_htcx/role/handheld_object
|
|
|
+ * - /user/vive_tracker_htcx/role/left_foot
|
|
|
+ * - /user/vive_tracker_htcx/role/right_foot
|
|
|
+ * - /user/vive_tracker_htcx/role/left_shoulder
|
|
|
+ * - /user/vive_tracker_htcx/role/right_shoulder
|
|
|
+ * - /user/vive_tracker_htcx/role/left_elbow
|
|
|
+ * - /user/vive_tracker_htcx/role/right_elbow
|
|
|
+ * - /user/vive_tracker_htcx/role/left_knee
|
|
|
+ * - /user/vive_tracker_htcx/role/right_knee
|
|
|
+ * - /user/vive_tracker_htcx/role/waist
|
|
|
+ * - /user/vive_tracker_htcx/role/chest
|
|
|
+ * - /user/vive_tracker_htcx/role/camera
|
|
|
+ * - /user/vive_tracker_htcx/role/keyboard
|
|
|
+
|
|
|
+
|
|
|
+Some final words
|
|
|
+----------------
|
|
|
+
|
|
|
+Hand tracking is an area that is still under active development and we are expecting improvements in the near future.
|
|
|
+
|
|
|
+The underlying principle that we're hoping will eventuate is that the action map will be used to handle interactions,
|
|
|
+while the hand tracking extension will primarily be a means for visualising the players hand.
|
|
|
+The hope here is that improvements to the OpenXR specification will ensure better portability between platforms.
|
|
|
|