desktop-xr-hands

Overview

Simulates WebXR Hand Tracking on desktop, using Mediapipe + Webcam.

Schema

system

The following properties can be set on the <a-scene> like this:

<a-scene desktop-xr-hands="cameraFov: 50; zOffset: 2"></a-scene>

| Property | Description | Default | | --------- | ------------------------------------------------------------ | ------- | | cameraFov | fov (width) of the webcam. Used in computing distance of hands from the camera. | 60 | | zOffset | z offset at which to render the hands. Hands will be level with the scene camera when they are approximately this distance from the webcam. | 1 |

per-hand

There are currently no configurable properties per-hand. However it is essential to include the desktop-xr-hands component on any entity that makes use of the WebXR Hand Tracking API.

Installation

<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist/desktop-xr-hands.min.js"></script>

Or via npm

npm install aframe-desktop-xr-hands

then in your code...

if (!AFRAME.components['desktop-xr-hands']) require('aframe-desktop-xr-hands')

(the if test avoids conflicts with other npm packages that may also use aframe-desktop-xr-hands)

Usage

Add the desktop-xr-hands attribute to any entity that uses hand-tracking-controls, or any other component that uses the WebXR Hand Tracking API (e.g. aframe-handy-controls).

The desktop-xr-hands attribute must be set on the entity before the component that uses the WebXR Hand Tracking API.

E.g.

<a-entity desktop-xr-hands hand-tracking-controls="hand: left"></a-entity>

Examples

desktop-xr-hands.html - The A-Frame Showcase Web XR Hand Tracking Example, with desktop-xr-hands added.

See also:

• Simple test page with mesh hand model

• Simple test page with dots hand model

Code

desktop-xr-hands

Limitations / Futures

This is an experimental component, with limited testing so far, and plenty of limitations, including:

• Not possible to enable for just one hand in a scene

• No option to toggle video feed & overlay on/off or control size / position

• No variation in reported radii of joints - all reported as a fixed 1cm.

• Mapping for Mediapipe Hand Model to WebXR Hand Model is not perfect, can probably be improved.

• Position tracking is pretty poor. Worst of all is distance tracking, which is based on palm-size (as the hand landmarker itself only generates hand pose data, not hand position data), but seems rather unstable. Possible solution is to integrate the Mediapipe pose landmarker which might provide better tracking of hand positions, than is possible through the hand landmarker alone.

• Hands are positioned in world space, relative to an assumed camera position of (0, 1.6, 0). If the camera is moved, hands don't move with it.

• Gesture detection is untested. Looking at Mediapipe fingertip positions for a pinch, it looks as though there could be precision issues with getting a reliable pinch.

• Only tested with A-Frame hand-tracking-controls so far. Would be nice to test with other hand tracking component systems, e.g. hand-tracking-controls-extras and handy-work.

• Most of the code in these components is not A-Frame-specific. Would be nice (and probably not too hard?) to abstract the code so it can be framework-agnostic and work with any THREE.js app, Babylon etc.

PRs for any of the above would be welcome!

References

Mediapipe reports positions for the following hand landmarks:

source

The WebXR API has to report positions & orientations for the following joints:

source

desktop-xr-hands takes care of mapping Mediapipe reported positions, to WebXR API positions and orientations.

The following parameters are set for each WebXR joint. The values are hard-coded within the desktop-xr-hands component.

| parameter | description | | | -------------------------- | ------------------------------------------------------------ | ---- | | position1 | The index of a mediapipe landmark to use for the position of this joint | | | position2 | The index of a second mediapipe landmark to use for the position of this joint | | | weight | The weight to attribute to position2 in positioning the joint. 0 => use position1, 1 => use position2, 0.5 => use the midpoint. | | | orientation start | The index of a mediapipe landmark to use for the joint orientation. This indicates the start point of the direction vector. For example, the wrist orientation is a vector from point 0 (start) to point 9 (end) | | | orientation end | The index of a mediapipe landmark to use for the joint orientation. This indicates the end point of the direction vector. | | | perpendicular vector start | (optional) The index of a mediapipe landmark to use for the joint orientation. This indicates the start point for a vector that the joint should be perpendicular to. Specifying a perpendicular vector is important to get the correct twist around the main orientation. | | | perpendicular vector end | (optional) The index of a mediapipe landmark to use for the joint orientation. This indicates the end point for a vector that the joint should be perpendicular to. | |