Augnito Speech SDK

You can use Augnito SDK to make use of the Speech Recognition AI. To get access credentials or talk to us about your project, get in touch with us at [email protected].

Installation

Install the library in your Typescript project.

Typescript

    npm install augnitosdk

Vanilla JS

use https://cdn.jsdelivr.net/npm/augnitosdk@latest/dist/augnitosdk.js to import the library from npm using CDN url.

Basic Usage

1- Import the library

Typescript

import { Augnito } from 'augnitosdk';

Vanilla JS

import { Augnito } from 'https://cdn.jsdelivr.net/npm/augnitosdk@latest/dist/augnitosdk.js';

Note - All other steps are common for Vanilla JS and Typescript implementations.

2- Create the configuration file and instantiate Augnito Client

const augnitoConfig: AugnitoConfig = {
  server: '<your domain server>',
  accountCode: '<your accountcode>',
  accessKey: '<your accesskey>',
  userTag: '<your usertag>',
  sourceApp: '<your sourceapp>',
  lmId: '<your lmid>'
  enableLogs: false, // Optional parameter, to check debug logs set to true
  listOfControls: [], // Optional parameter, set a list of controlId and controlName for SDK to handle multiple text field navigation commands
  editor: EditorType.HTML // Optional parameter, set EditorType enum for SDK to insert sppech-to-text display at cursor placed inside field and to handle commands dictated. Other enum values are EditorType.CKEDITOR4 | EditorType.CKEDITOR5 | EditorType.GENERIC,
};

const augnito = new Augnito(augnitoConfig);

3- Toggle the client!

Now all you have to do is toggle the status when you want to connect/disconnect!

// Toggles the Speech API connection. Set optional parameter shouldPauseResume to true to toggle pause and resume dictation. This will ensure socket connection remains connected throughout. If no parameter is given, then the method will function as toggle of start and stop dictation. This will create a new socket connection on every start
augnito.toggleListening((shouldPauseResume = false));

augnito.requestMicOff(); //call this method to close the socket connection and clear all resources in case shouldPauseResume is set to true

Mobile Application

Whether you want to use the Augnito Mic as your microphone, or you want to develop your own app, we got you covered.

All you have to do is initialize the mobile configuration. The SDK will generate a value that you need to use to generate a QR code which should be scanned in the App.

import { AugnitoMobileConfig } from 'augnitosdk';

const mobileConfig: AugnitoMobileConfig = {
  enableLogs: false
};
const augnitoMobile = augnito.initMobileClient(mobileConfig);

//Generate a QR Code with the following value
const code = augnito.getQRCode();

Config Philips Speech Mic

If you have a Philips speech mic and want to use it as the input device, then include the below code.

import { PhilipsMicMode } from 'augnitosdk';

const philipMicConfig = {
  philipsMicMode: PhilipsMicMode['HandsFree'] //other options include PhilipsMicMode.PushToTalk or PhilipsMicMode.BOTH
};
const philipMicHandler = augnito.initPhilipMicClient(philipMicConfig);

• HandsFree - where you push the record button speak and the push the button to stop
• PushToTalk - where you press and hold the record button to speak. Release of the button will stop
• BOTH - SDK will intelligently handle the other two modes based on user action

Augnito - Speech Related Callbacks

| Name | type | Description | | ------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | onFinalResult | (finalText: Recipe) => boolean | A final result is called when text has been processed by the Speech Server. Returns true to stop the processing in the SDK | | onCommandResult | (command: Recipe) => boolean | A command generated from the server. Returns true to stop the processing in the SDK | | onStateChanged | (isConnected: boolean) => void | Callback to indicate the status of the connection has changed. | | onSessionEvent | (data: AugnitoSocketResponse) => void | Callback to intercept Session Events | | onIdleMic | () => void | Callback when the Mic is idle for 5 minutes | | onMicrophoneOnError | (isWebClient: boolean) => void | Callback triggered when trying to start the connection, but it fails because it is already opened. (It can be opened in another tab, or it was not closed properly) | | onError | (error: AugnitoSDKErrorMessage) => void | Callback triggered when the SDK encouters an error |

Augnito Mobile - Speech Related Callbacks

| Name | type | Description | | ------------------- | ---------- | ------------------------------------------------------ | | onConnectionRequest | () => void | Callback when a mobile client is requesting to connect | | onMobileScan | () => void | Callback when a mobile client has scanned the QR Code |

Philips Speech Mic Related Callbacks

| Name | type | Description | | ------------------- | ---------- | ------------------------------------------------------------ | | onPhilipsMicPressed | () => void | Callback when record button is pressed on Philips Speech Mic |

Augnito Methods

| prop | type | notes | | ------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | toggleListening | void | Toggles the Speech API connection: if already connected, it will stop the web socket; otherwise, it will start a new connection. If optional parameter shouldPauseResume is set to true then listening to mic will be suspended and resumed, here web socket will remain alive throughout the session. Consumers of the sdk are expected to call requestMicOff to stop the web sockeet and clean the resources in the end | | isConnected | boolean | Returns the status of the web socket: true if connected; otherwise false. | | getQRCode | string | Returns the Generated Value for the QR Code (If and only if the DeviceId was provided). | | dispose | void | Dispose all resources. | | initMobileClient | AugnitoMobile | Creates a web socket for getting notifications from client mobile application: only to be used with Mobile App | | apisFactory | FactoryAPI | Returns an Abstract Factory to create different API Clients. | | requestMicOff | void | Sends a request to close all opened connections. | | initPhilipMicClient | PhilipsMicClient | To be used with Philips Speecch Mic. It will listen to press event of record button on the mic. |

MacrosAPI

MacrosAPI Methods

| name | type | notes | | ----------- | -------------------- | -------------------------------------------------- | | getMacros | Promise: MacroResult | Returns a list of macros for the specified user. | | upsertMacro | Promis: MacroResult | Creates or Updates a Macro for the specified user. | | deleteMacro | Promise: MacroResult | Deletes a Macro for the specified user. |

1- Create Macros Client

// If you already have an augnito client..
const macrosClient = augnito.apisFactory.getMacrosAPI();

// OR
import { MacrosAPI } from 'augnitosdk';
const macrosClient = new MacrosAPI(accountCode, accessKey, server);

2- Use the Macros Client to list, create, update or delete user macros

macrosClient
  .getMacros({
    UserTag: userTag
  })
  .then((result) => {
    console.log(result);
  });

Formatting Preference API

Formatting Preference API Interfaces

interface Edge
{
  PK: string;
  SK: string;
}

interface UserFormattingRequest
{
  PageId?: Edge; // Optional page edge for next and previous pages of search results
  Limit?: number; // Optional limit for number of search results
}

interface FormattingItem
{
  SpokenForm: string; // The spoken form of the formatting preference
  DisplayForm: string; // The display form of the formatting preference
  ID: string; // The ID of the formatting preference
}

interface FormattingResult
{
  Status: number; // Response status code
  ErrorMessage: string; // Error message if any
  Data: {
    StatusMessage: string; // Response status message
    FormattingPreferenceItems? :
    {
        PageSize: number; // Number of items in the current page
        PageIdNext?: Edge; // Next page edge if any
        PageIdPrevious?: Edge; // Previous page edge if any
        Items: Array<FormattingItem>; // List of formatting preferences
    };
  };
}

interface UpsertFormattingRequest {
  SpokenForm: string; // The spoken form of the formatting preference
  DisplayForm: string; // The display form of the formatting preference
  ID?: string; // The ID of the formatting preference if updating an existing preference
}

RemoveFormattingRequest {
  ID: string; // The Id of the formatting preference to be deleted
}

Formatting Preference API Methods

| name | type | notes | | -------------------------------------------------- | ------------------------- | ------------------------------------------------------------------ | | getFormatting(request: UserFormattingRequest) | Promise: FormattingResult | Returns a list of formatting preferfences for the specified user. | | upsertFormatting(request: UpsertFormattingRequest) | Promise: FormattingResult | Creates or Updates a formatting preference for the specified user. | | removeFormatting(request: RemoveFormattingRequest) | Promise: FormattingResult | Deletes a formatting preference for the specified user. |

1- Create Formatting Preference Client

// If you already have an augnito client..
const formattingAPI = augnito.apisFactory.getFormattingAPI();

// OR
import { FormattingAPI } from 'augnitosdk';
const formattingAPI = new FormattingAPI({
  UserCode: <your usercode>,
  SubscriptionCode: <your subscription code>,
  AccessKey: <your access key>,
  SmId: <your sm id>,
  DeviceCode: <your device code>,
  SourceApp: <your source app>,
  AccountCode: <your account code>,
  Server: <your  server>,
  EnableLogs: true | false,
});

2- Use the Formatting Preference Client to list, create, update or delete user macros

formatApi
  .getFormatting({
    Limit: 10
  })
  .then((result) => {
    console.log('formatting', result);
  });

Vocabulary API

Vocabulary items can be added to the model for a specific user and model. When the word has been added and is ready to be recognised a notification will be sent to the client and the microphone will need to be restarted.

Vocabulary Models

interface Edge
{
  PK: string;
  SK: string;
}

interface UserVocabularyRequest
{
  PageId?: Edge; // Optional page edge for next and previous pages of search results
  Limit?: number; // Optional limit for number of search results
}

interface VocabularyItem
{
  Word: string; // The word
  SpokenWord: string | null; // The , optional, spoken form of the word
  IsAbbreviation: boolean; // Whether the word is an abbreviation or not
  CreatedAt: Date; // The date the word was added to the vocabulary
  Status: string; // The status of the word
}

interface VocabularyResult
{
  Status: number; // Response status code
  ErrorMessage: string; // Error message if any
  Data: {
    StatusMessage: string; // Response status message
    WordItems? :
      {
        SubscriptionCode: string; // The subscription code
        UserCode: string; // The user code the word has been added for
        PageSize: number; // Number of items in the current page
        PageIdNext?: Edge; // Next page edge if any
        PageIdPrevious?: Edge; // Previous page edge if any
        Items: Array<VocabularyItem>; // List of vocabulary items
      };
  };
}

interface UpsertVocabularyRequest
{
  file: File; // Audio file of the word to be added
  Word: string; // The word
  SpokenWord: string; // The, optional, spoken form of the word
  IsAbbreviation: boolean; // Whether the word is an abbreviation or not
}

interface RemoveVocabularyRequest
{
  Word: string; // The word to be removed
}

VocabularyAPI Methods

| name | type | notes | | -------------------------------------------------- | ------------------------- | ----------------------------------------------------- | | getVocabulary(request: UserVocabularyRequest) | Promise: VocabularyResult | Returns a list of vocabulary for the specified user. | | upsertFormatting(request: UpsertVocabularyRequest) | Promise: VocabularyResult | Creates or Updates vocabulary for the specified user. | | removeFormatting(request: RemoveVocabularyRequest) | Promise: VocabularyResult | Deletes a vocabulary for the specified user. |

VocabularyAPI Callback

| Name | type | Description | | ----------------- | ---------------------------------------------- | ------------------------------------------------------- | | onVocabularyAdded | (notification: AugnitoAppNotification) => void | Notification that a vocabulary word has been processed. |

Vocbaulary Helper Methods

| Name | type | Description | | ---------------- | ---------------------------- | ------------------------------------------------------------ | | startRecording() | (Promise) => void | Starts recording audio. Will turn off main Augnito mic if on | | stopRecording() | (blob:Promise) => void | Stops recording audio and returns the audio |

The audio for the vocabulary to be added must be between 1 and 8 seconds long. If you call stop recording less than 1 second after start the recording will wait for a second to have passed since it started before resolving the promise. If you call stop recording more than 8 seconds after start the returned recording will be the first 8 seconds recorded.

1- Create Vocabulary Client

const vocabularyAPI = augnito.apisFactory.getVocabularyAPI();

// Add a callback to be notified when a vocabulary word has been processed
vocabularyAPI.onVocabularyAdded = (notification: AugnitoAppNotification) => {
  console.log(notification);
};

2 - Add a new vocabulary item

// Start recording
vocabularyAPI.startRecording();
... // Record for 1-8 seconds
// Stop recording
vocabularyAPI.stopRecording().then((blob) => {
  // Add the word to the vocabulary
  vocabularyAPI.upsertVocabulary({
    file: new File([blob], 'test.wav'),
    Word: 'test',
    SpokenWord: 'test',
    IsAbbreviation: false,
  });
});

3 - List vocabulary items from multiple pages

const loadWords = (page?: Edge) => {
    vocabularyAPI
      .getVocabulary({ Limit: 10, PageId: page })
      .then(function (result) {
        if (typeof result != 'string') {
          result.Data.WordItems?.Items.forEach((value) => {
            console.log(value);
          });
          if (result.Data.WordItems?.PageIdNext) {
            loadWords( result.Data.WordItems.PageIdNext);
          }
        }
      });
  };

  loadWords();

MedicineOrderingAPI

MedicineOrderingAPI Methods

| Name | type | Description | | -------------------- | ----------------------------------------- | -------------------------------------------------------------- | | getMedicineOrderData | Promise: MedicineOrderingResult Or String | Returns medicine ordering data for text specified by the user. |

1- Create MedicineOrdering Client

// If you already have an augnito client..
const orderingClient = augnito.apisFactory.getMedicineOrderingAPI();

2- Use the MedicineOrdering Client to get medicine order data in the following manner

orderingClient
  .getMedicineOrderData({
    Text: ''
  })
  .then((result) => {
    console.log(result);
  });