Picovoice SDK for React Native

Picovoice

Made in Vancouver, Canada by Picovoice

Picovoice is an end-to-end platform for building voice products on your terms. It enables creating voice experiences similar to Alexa and Google. But it entirely runs 100% on-device. Picovoice is

• Private: Everything is processed offline. Intrinsically HIPAA and GDPR-compliant.
• Reliable: Runs without needing constant connectivity.
• Zero Latency: Edge-first architecture eliminates unpredictable network delay.
• Accurate: Resilient to noise and reverberation. It outperforms cloud-based alternatives by wide margins *.
• Cross-Platform: Design once, deploy anywhere. Build using familiar languages and frameworks.

Compatibility

This binding is for running Picovoice on React Native 0.62.2+ on the following platforms:

• Android 5.0+ (API 21+)
• iOS 13.0+

Installation

To start installation be sure you have installed yarn and cocoapods. Then add these native modules to your React Native project.

yarn add @picovoice/react-native-voice-processor
yarn add @picovoice/porcupine-react-native
yarn add @picovoice/rhino-react-native
yarn add @picovoice/picovoice-react-native

or

npm i @picovoice/react-native-voice-processor
npm i @picovoice/porcupine-react-native
npm i @picovoice/rhino-react-native
npm i @picovoice/picovoice-react-native

Link the iOS packages:

cd ios && pod install && cd ..

NOTE: Due to a limitation in React Native CLI auto-linking, these native modules cannot be included as transitive dependencies. If you are creating a module that depends on these packages you will have to list these as peer dependencies and require developers to install them alongside.

AccessKey

Picovoice requires a valid Picovoice AccessKey at initialization. AccessKey acts as your credentials when using Picovoice SDKs. You can get your AccessKey for free. Make sure to keep your AccessKey secret. Signup or Login to Picovoice Console to get your AccessKey.

Permissions

To enable recording with the hardware's microphone, you must first ensure that you have enabled the proper permission on both iOS and Android.

On iOS, open your Info.plist and add the following line:

<key>NSMicrophoneUsageDescription</key>
<string>[Permission explanation]</string>

On Android, open your AndroidManifest.xml and add the following line:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.INTERNET" />

Usage

The module provides you with two levels of API to choose from depending on your needs.

High-Level API

PicovoiceManager provides a high-level API that takes care of audio recording. This class is the quickest way to get started.

The static constructor PicovoiceManager.create will create an instance of a PicovoiceManager using a Porcupine keyword file and Rhino context file that you pass to it.

const accessKey = "${ACCESS_KEY}" // obtained from Picovoice Console (https://console.picovoice.ai/)

try {
    _picovoiceManager = await PicovoiceManager.create(
        accessKey
        '/path/to/keyword.ppn',
        wakeWordCallback,
        '/path/to/context.rhn',
        inferenceCallback);
} catch (e) { }

To use wake word (.ppn) and context (.rhn) files in your React Native application you'll need to add the files to your platform projects. Android models must be added to ./android/app/src/main/assets/, while iOS models can be added anywhere under ./ios, but must be included as a bundled resource in your iOS (i.e. add via XCode) project. The paths used as initialization arguments are relative to these device-specific directories.

Alternatively, if the files are deployed to the device with a different method, the absolute paths to the files on device can be used.

The wakeWordCallback and inferenceCallback parameters are functions that you want to execute when a wake word is detected and when an inference is made.

wakeWordCallback() {
    // wake word detected!
}

inferenceCallback(inference) {
    if (inference.isUnderstood) {
        // do something with:
        // inference.intent - string representing intent
        // inference.slots - Object<string, string> representing the slot values
    }
}

Picovoice accepts the following optional parameters:

• porcupineModelPath: path to file containing model parameters for the wake word engine
• porcupineSensitivity: overrides the default wake word sensitivity
• rhinoModelPath: path to file containing model parameters for the speech-to-intent engine
• rhinoSensitivity: overrides the default inference sensitivity
• processErrorCallback: called if there is a problem encountered while processing audio
• endpointDurationSec: sets how much silence is required after a spoken command
• requireEndpoint: indicates whether Rhino should wait for silence before returning an inference

const accessKey = "${ACCESS_KEY}" // obtained from Picovoice Console (https://console.picovoice.ai/)

try {
    _picovoiceManager = PicovoiceManager.create(
                accessKey,
                '/path/to/keyword.ppn',
                wakeWordCallback,
                '/path/to/context.rhn',
                inferenceCallback,
                processErrorCallback,
                porcupineSensitivity,
                rhinoSensitivity,
                "/path/to/porcupine_model.pv",
                "/path/to/rhino_model.pv",
                endpointDurationSec,
                requireEndpoint);
} catch (e) { }

Once you have instantiated a PicovoiceManager, you can start audio capture and processing by calling:

try {
    await _picovoiceManager.start();
} catch (e) { }

And then stop it by calling:

try {
    await _picovoiceManager.stop();
} catch (e) { }

Finally, once you no longer need the PicovoiceManager, be sure to explicitly release the resources allocated to it:

await _picovoiceManager.delete();

With PicovoiceManager, the @picovoice/react-native-voice-processor module handles audio capture and automatically passes it to the inference engine.

Low-Level API

Picovoice provides low-level access to the Picovoice platform for those who want to incorporate it into an already existing audio processing pipeline.

Picovoice is created by passing a Porcupine keyword file (.ppn) and Rhino context file (.rhn) to the create static constructor. Sensitivity, model files, endpointDurationSec and requireEndpoint are optional.

const accessKey = "${ACCESS_KEY}" // obtained from Picovoice Console (https://console.picovoice.ai/)

async createPicovoice() {
    let porcupineSensitivity = 0.7
    let rhinoSensitivity = 0.6
    let endpointDurationSec = 1.5
    let requireEndpoint = false

    try {
        _picovoice = await Picovoice.create(
            accessKey,
            '/path/to/keyword/file.ppn',
            wakeWordCallback,
            '/path/to/context/file.rhn',
            inferenceCallback,
            porcupineSensitivity,
            rhinoSensitivity,
            "/path/to/porcupine/model.pv",
            "/path/to/rhino/model.pv",
          endpointDurationSec,
            false)
    } catch (e) { }
}

wakeWordCallback() {
    // wake word detected!
}

inferenceCallback(inference) {
    if (inference.isUnderstood) {
        // do something with:
        // inference.intent - string representing intent
        // inference.slots - Object<string, string> representing the slot values
    }
}

To use Picovoice, just pass frames of audio to the process function. The callbacks will automatically trigger when the wake word is detected and then when the follow-on command is detected.

let buffer = getAudioFrame();

try {
    await _picovoice.process(buffer);
} catch (e) { }

For process to work correctly, the audio data must be in the audio format required by Picovoice. The required audio format is found by calling .sampleRate to get the required sample rate and .frameLength to get the required frame size. Audio must be single-channel and 16-bit linearly-encoded.

Finally, once you no longer need the Picovoice, be sure to explicitly release the resources allocated to it:

await _picovoice.delete();

Custom Wake Word & Context Integration

To add custom wake words and contexts to your React Native application you'll need to add the model files to your platform projects.

Adding Android Models

Android models must be added to ./android/app/src/main/assets/.

Adding iOS Models

iOS models can be added anywhere under ./ios, but it must be included as a bundled resource. The easiest way to include a bundled resource in the iOS project is to:

1. Open XCode.
2. Either:

• Drag and Drop the model/keyword file to the navigation tab.
• Right-click on the navigation tab, and click Add Files To ....

This will bundle your models together when the app is built.

Using Custom Models

Pass the file paths (relative to the assets/resource) directory:

const accessKey = "${ACCESS_KEY}"; // obtained from Picovoice Console (https://console.picovoice.ai/)

let wakeWordPath = '';
let contextPath = '';

if (Platform.OS == 'android') {
    wakeWordPath = `keyword_android.ppn`;
    contextPath = `context_android.rhn`;
} else if (Platform.OS == 'ios') {
    wakeWordPath = `keyword_ios.ppn`;
    contextPath = `context_ios.rhn`;
}

Alternatively, if the model files are deployed to the device with a different method, the absolute paths to the files on device can be used.

Non-English Models

In order to detect wake words and run inference in other languages you need to use the corresponding model file. The model files for all supported languages are available here and here.

Demo App

Check out the Picovoice React Native demo to see what it looks like to use Picovoice in a cross-platform app!