Android - RNVeyetalsNative

RNVeyetalsNative component is an Android only component to scan user vitals using smartphone camera.

General architecture of the app using RNVeyetalsNative component is as follows:

Following are the steps to integrate RNVeyetalsNative in your react native application.

Install RNVeyetalsNative NPM package

npm i react-native-veyetals-native

Make sure to use version 2.0.5 and above.

Update Android gradle files to include dependencies

Project level build.gradle file

In project level build.gradle file update core components version as follows:

kotlin_version = "1.9.0"

Update Android SDK versions as follows:

minSdkVersion = 24
compileSdkVersion = 33
targetSdkVersion = 33

buildscript repositories object must include following repositories:

google()
jcenter()
mavenCentral()
maven { url "https://jitpack.io" }

buildscript dependencies object must include following dependencies:

classpath("com.android.tools.build:gradle:7.2.1")
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"

allprojects repositories object must include following repositories:

google()
mavenLocal()
jcenter()

App level build.gradle file

Apply following kotlin plugins:

apply plugin: 'kotlin-android'
apply plugin: 'kotlin-kapt'
apply plugin: 'kotlin-android-extensions'

Enable databinding in android object:

buildFeatures {
    dataBinding true
}

Update compile option to Java 11:

compileOptions {
  sourceCompatibility JavaVersion.VERSION_11
  targetCompatibility JavaVersion.VERSION_11
}

Update Kotlin JVM target to 1.8:

kotlinOptions {
    jvmTarget = '11'
}

Add dependencies in dependencies object:

//lifecycle
implementation 'androidx.lifecycle:lifecycle-runtime-ktx:2.2.0'
implementation 'androidx.lifecycle:lifecycle-extensions:2.2.0'


//Coroutines
implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.7'
implementation 'androidx.lifecycle:lifecycle-livedata-ktx:2.2.0'

//CameraX
implementation "androidx.camera:camera-core:1.1.0-alpha08"
implementation "androidx.camera:camera-camera2:1.1.0-alpha08"
implementation "androidx.camera:camera-lifecycle:1.1.0-alpha08"
implementation "androidx.camera:camera-view:1.0.0-alpha28"

//ML
implementation 'com.google.mlkit:face-detection:16.1.2'
implementation "org.tensorflow:tensorflow-lite:2.2.0"
implementation "org.tensorflow:tensorflow-lite-gpu:2.2.0"
implementation 'org.tensorflow:tensorflow-lite-support:0.0.0-nightly'

//Pytorch
implementation 'org.pytorch:pytorch_android_lite:1.13.1'
implementation 'org.pytorch:pytorch_android_torchvision_lite:1.13.1'

//Retrofit
implementation "com.squareup.okhttp3:okhttp:4.7.2"
implementation "com.squareup.retrofit2:retrofit:2.9.0"
implementation "com.squareup.okhttp3:logging-interceptor:4.9.0"
implementation "com.squareup.retrofit2:converter-gson:2.9.0"

//Shared preferences
implementation 'com.pixplicity.easyprefs:EasyPrefs:1.10.0'

gradle-wrapper.properties

Change gradle plugin URL as follows

distributionUrl=https\://services.gradle.org/distributions/gradle-7.3.3-all.zip

settings.gradle

Add following code in settings.gradle file in case it's not already included by React Native.

include ':markitech-opencv'
project(':markitech-opencv').projectDir = new File(rootProject.projectDir, '../node_modules/markitech-opencv/android')
include ':markitech-veyetals-sdk'
project(':markitech-veyetals-sdk').projectDir = new File(rootProject.projectDir, '../node_modules/markitech-veyetals-sdk/android')

How to use RNVeyetalsNative component in RN app

Integrating RNVeyetalsNative component in ReactNative application is very straight forward and it requires very little input from the client application. Following code is all that client application requires to integrate RNVeyetalsNative component.

import { RNVeyetalsNative } from 'react-native-veyetals-native';

const customLabels = {
  multiCameraAlertTitle: "Multi Camera Alert Title",
  multiCameraAlertMessage: "Multi Camera Alert Message",
  userMovementErrorMessage: "User Movement Error Message",
  faceAlignmentMessage: "Face Alignment Message",
  //Custom label for invalid vital alert view title
  INVALID_VITAL_ALERT_TITLE: "Invalid vital value",
  //Custom labels for invalid vitals captured via face scan
  INVALID_RPPG_HR_ALERT_MSG: "Invalid heart rate value",
  INVALID_RPPG_HRV_ALERT_MSG: "Invalid heart rate variability value",
  INVALID_RPPG_STRESS_ALERT_MSG: "Invalid stress level value",
  INVALID_RPPG_RL_ALERT_MSG: "Invalid respiratory rate value",
  INVALID_RPPG_BP_ALERT_MSG: "Invalid blood pressure value",
  INVALID_RPPG_O2_ALERT_MSG: "Invalid oxygen saturation value",
  //Custom labels for invalid vitals captured via finger scan
  INVALID_PPG_HR_ALERT_MSG: "Invalid heart rate value"
  INVALID_PPG_HRV_ALERT_MSG: "Invalid heart rate variability value",
  INVALID_PPG_STRESS_ALERT_MSG: "Invalid stress level value",
  INVALID_PPG_RL_ALERT_MSG: "Invalid respiratory rate value",
  INVALID_PPG_BP_ALERT_MSG: "Invalid blood pressure value",
  INVALID_PPG_O2_ALERT_MSG: "Invalid oxygen saturation value"
}
//custom label values are optional. SDK uses default value if corresponding label value is not found in the object

try {
      const vitals = await RNVeyetalsNative.start(<USER_ID>, <CHANNEL_ID>, mode, isLiteVersion, showUI, customLabels);
        if (vitals !== undefined) {
          console.log(vitals);
          /*
          {
            "Heart-rate": 64 (BPM),
            "HRV": 100 (ms),
            "02-saturation": 97 (%),
            "Respiration-rate": 12 (RPM),
            "Systolic": 112 (mmHg),
            "Diastolic": 70 (mmHg),
            "Stress-levels": [-1:Relaxed | 0:Normal | 1:Low Stress | 2:Medium Stress | 3.0:High Stress | 4.0:Very High Stress],
            "snr": 3.0,
            "glucose": 104 (mg/dL),
            "emotion": [HAPPY | SAD | ANGRY | CONFUSED | DISGUSTED | SURPRISED | CALM | UNKNOWN | FEAR],
            "isValidHeartRate": "true",
            "isValidBloodPressure": "true"
          }
          */
        }
      } catch (e) {
        console.log(e);
      }

RNVeyetalsNative fully encapsulate the vitals scanning process. RNVeyetalsNative.start() command launches a new Android activity and whole scanning processing takes places within this activity. At the end of the process, client application either get final vitals from the component or it throws an error. Intermediate vitals are not retured to the client application.

Mode

Following scanning modes are supported:

face: Get vitals by scanning face. finger: Get vitals by scanning finger.

Lite Version

RNVeyetalsNative also supports lite version whereby only Heart Raten, Heart Rate Variability and stress levels are scanned. Opt for lite version by passing true value for isLiteVersion parameter.

showUI

Using this flag host application can enable or disable vitals UI.

Errors

RNVeyetalsNative component may throw following errors:

E_ACTIVITY_DOES_NOT_EXIST: RNVeyetalsNative component couldn't initialize internal activity.
E_VEYETALS_CANCELLED: Process is cancelled by the user.
E_FAILED_TO_INITIALIZE_VEYETALS: Failed to initialize RNVeyetalsNative component. E_NO_FINAL_RESULTS_FOUND: Component could not generate final vitals for client app.

iOS - VeyetalsViewIOS

VeyetalsViewIOS is iOS only ReactNative view to scan user vitals using iPhone Camera.

NPM Package instructions are the same as for Android. Following are the system requirements for integration of VeyetalsViewIOS in React Native iOS application.

iOS Version: 15.0 and above.

Swift Version: 5.0

Camera Usage Permission: Add camera usage string in application's info.plist file

How to use VeyetalsViewIOS component in iOS app

import { VeyetalsViewIOS } from 'react-native-veyetals-native';

async function onCompleted(vitals) {
  console.log('Veyetals scanning is complete');
  //process vitals - sample output
  /*
    {
      "Heart-rate": "64",
      "HRV": "100",
      "02-saturation": "97",
      "Respiration-rate": "12",
      "Systolic": "112",
      "Diastolic": "70",
      "Stress-levels": "-1",
      "snr": "3",
      "glucose": "104",
      "emotion": "Calm",
      "isValidHeartRate": "true",
      "isValidBloodPressure": "true"
    }
  */
}

async function onCancelled() {
  console.log('Veyetals scanning cancelled');
  //update view
}

const customLabels = {
  multiCameraAlertTitle: "Multi Camera Alert Title",
  multiCameraAlertMessage: "Multi Camera Alert Message",
  userMovementErrorMessage: "User Movement Error Message",
  faceAlignmentMessage: "Face Alignment Message",
  //Custom label for invalid vital alert view title
  INVALID_VITAL_ALERT_TITLE: "Invalid vital value",
  //Custom labels for invalid vitals captured via face scan
  INVALID_RPPG_HR_ALERT_MSG: "Invalid heart rate value",
  INVALID_RPPG_HRV_ALERT_MSG: "Invalid heart rate variability value",
  INVALID_RPPG_STRESS_ALERT_MSG: "Invalid stress level value",
  INVALID_RPPG_RL_ALERT_MSG: "Invalid respiratory rate value",
  INVALID_RPPG_BP_ALERT_MSG: "Invalid blood pressure value",
  INVALID_RPPG_O2_ALERT_MSG: "Invalid oxygen saturation value",
  //Custom labels for invalid vitals captured via finger scan
  INVALID_PPG_HR_ALERT_MSG: "Invalid heart rate value"
  INVALID_PPG_HRV_ALERT_MSG: "Invalid heart rate variability value",
  INVALID_PPG_STRESS_ALERT_MSG: "Invalid stress level value",
  INVALID_PPG_RL_ALERT_MSG: "Invalid respiratory rate value",
  INVALID_PPG_BP_ALERT_MSG: "Invalid blood pressure value",
  INVALID_PPG_O2_ALERT_MSG: "Invalid oxygen saturation value"
}

<VeyetalsViewIOS
  style={{
    flex: 1,
  }}
  config={{
    userId: <USER_ID>,
    channelId: <CHANNEL_ID>,
    mode: [face|finger],
    isLightWeight: false,
    showUI: true,
    customLabels: customLabels,
  }}
  onComplete={onCompleted}
  onCancel={onCancelled}
/>

User Management

RNVeyetalsComponent internally manages users to keep track of service usage. For this purpose it requires following 3 pieces of information from the client app:

CHANNEL_ID: Uniquely identifies an organisation. SDK clients must request CHANNEL_ID from Veyetals team before integrating the component in their application.
APP_ID: Uniquely identifies an app. A channel may have multiple apps using the RNVeyetalsNative component / VeyetalsViewIOS view. APP_ID is programatically obtained from the client application.
USER_ID: Uniquely identifies an app user for a specific APP_ID and for a specific CHANNEL_ID. Used to track users per app per channel.

These three pieces of information help Veyetals to aggregate the number of users using Veyetals service per app per channel and help generate billing. Before integrating SDK it’s essential for client to contact Veyetals team and activate CHANNEL_ID and APP_ID.

Data Privacy

RNVeyetalsNative component doesn’t store users' video or vitals data; neither on the device nor in the cloud. Therefore, it’s responsibility of the client application to protect user data.

Stress Mappings

-1 --> Relaxed
0  --> Normal
1  --> Low Stress
2  --> Medium Stress
3  --> High Stress
4  --> Very High Stress
5 --> Extremely High Stress