mediapipe-face-mesh extension

Overview

This extension provides online tracking of facial posture during trials using the MediaPipe Face Mesh library. It currently records the estimated 3D translation and rotation.

The primary purpose of this extension is to use it in plugins. For instance, it allows creation of interactive virtual mirror paradigms similar to the Open Virtual Mirror Framework. However, it can also be used as an online face tracker if recording of sensitive facial data can be avoided. Refer to trial parameters for more details.

Loading

In browser

<script src="https://unpkg.com/@jspsych-contrib/[email protected]">

Via NPM

npm install @jspsych-contrib/extension-mediapipe-face-mesh

import jsPsychExtensionMediapipeFacemesh from '@jspsych-contrib/extension-mediapipe-face-mesh';

Compatibility

This extension was developed for, and tested with jsPsych v7.3.0.

Documentation

Initialization Parameters

Initialization parameters can be set when calling initJsPsych()

initJsPsych({
  extensions: [
    {type: jsPsychExtensionMediapipeFacemesh, params: {...}}
  ]
})

Parameter | Type | Default Value | Description ----------|------|---------------|------------ locateFile | function | see source | The function that is used to locate the binary blobs used by the face_mesh tracker.

Trial Parameters

Trial parameters can be set when adding the extension to a trial object.

var trial = {
  type: jsPsych...,
  extensions: [
    {type: jsPsychExtensionMediapipeFacemesh, params: {...}}
  ]
}

Parameter | Type | Default Value | Description ----------|------|---------------|------------ record | boolean | false | Should the data generated as described below recorded or immediately discarded. Please note, that long recording might require lots of memory.

Data Generated

In addition to the default data collected by all plugins, this plugin collects all parameter data described above and the following data for each trial.

| Name | Type | Value | | ---------------- | ----------- | ---------------------------------------- | | frame_id | numeric | The ID of the frame from which the tracking data was obtained | | transformation | [numeric] | Array containing the full 3x4 transformation matrix for homogeneous coordinates. See MediaPipe doc for more details.| | rotation | [numeric] | Array containing the Euler rotation angles in XYZ order. | | translation | [numeric] | Array containing the translation along XYZ axes. |

Usage in other plugins

This extension allows creation of interactive experiments, e.g., virtual mirrors as in this project. Here is an example of the sage of the extension from another plugin:

  class MyInteractivePlugin {

    constructor(jsPsych) {
        this.jsPsych = jsPsych;
        this.mediapipe = this.jsPsych.extensions["mediapipe-face-mesh"];
    }

    trial(display_element, trial) {

        const callback = (trackingResult) => {
            console.log(trackingResult);
        };

        this.mediapipe.addTrackingResultCallback(callback);

        // Do some stuff

        this.jsPsych.pluginAPI.setTimeout(
            () => {
                this.mediapipe.removeTrackingResultCallback(callback);
                this.jsPsych.finishTrial();
            },
            trial["trial_duration"]);
    }