Slang Fitness Assistant

Try out the playground app for developers to understand the assistant.

Install the Slang Fitness Assistant package

yarn setup

If you use yarn for install packages, run the below command,

yarn add @slanglabs/slang-conva-react-native-fitness-assistant

npm setup

If you use npm for managing your packages, run the below command

npm install @slanglabs/slang-conva-react-native-fitness-assistant --save

Because Slang uses native libraries, you need to link the package to your codebase to run the automatic linking steps

react-native link @slanglabs/slang-conva-react-native-fitness-assistant

Android

Finally, add the path to the Slang maven repository (to download native library dependencies) to your top-level gradle file

# Add this to your top level gradle file

allprojects {  
    repositories {    
        …    
        maven { url "http://maven.slanglabs.in:8080/artifactory/gradle-release" }  
    }
}

iOS

Finally, add the path to the Slang cocoapod repository (to download native library dependencies) to your top-level pod file

# Add this to your top level podfile file

source 'https://github.com/SlangLabs/cocoapod-specs'
source 'https://github.com/CocoaPods/Specs.git'

Code Integration

Initialization

The next step is to initialize the SDK with the keys you obtained after creating the Assistant in the Slang console.

NOTE This should ideally be done in the componentDidMount of your main app component

import SlangFitnessAssistant from '@slanglabs/slang-conva-react-native-fitness-assistant';

SlangFitnessAssistant.initialize({
    requestedLocales: ['en-IN', 'hi-IN'], // The languages to enable
    assistantId: '<assistant id>',        // The Assistant ID from the console
    apiKey: '<API Key>',                  // The API key from the console
})

Show the Trigger (microphone icon)

NOTE One can call "show" and "hide" methods as required to control the visibility of the Assistant

SlangFitnessAssistant.ui.showTrigger() // There is a corresponding hideTrigger too if needed

NOTE The trigger is sticky, which means that it will show up on all Activities after it is made visible. To prevent the trigger from showing up on specific activities, you will need to call: SlangFitnessAssistant.ui.hideTrigger()

Implement Actions

Last but not the least, the app needs to implement the Actions associated with the various User Journeys supported by the Assistant. This can be done as shown below

const actionHandler = {
  onFoodLog: (foodInfo, foodLoggingUserJourney) => {
    // Handle the food log request    
    // ...
    
    return FoodLoggingUserJourney.FoodLoggingCompleteAppState(
        FoodLoggingUserJourney.AppStateCondition.SUCCESS
    )
    
  },

  onSugarLog: (sugarInfo, sugarLoggingUserJourney) => {
    // Handle the sugar log request    
    // ...
    
    return SugarLoggingUserJourney.SugarLoggingCompleteAppState(
        SugarLoggingUserJourney.AppStateCondition.SUCCESS
    )
    
  },

  onWeightLog: (weightInfo, weightLoggingUserJourney) => {
    // Handle the weight log request    
    // ...
    
    return WeightLoggingUserJourney.WeightLoggingCompleteAppState(
        WeightLoggingUserJourney.AppStateCondition.SUCCESS
    )
    
  },

  onAssistantError: errorCode => {
    // Handle errors that might have happened during the processing of the
    // Assistant
    
    // Error codes available 
    // FATAL_ERROR, SYSTEM_ERROR, ASSISTANT_DISABLED, INVALID_CREDENTIALS,    
  },
};

SlangFitnessAssistant.setAction(actionHandler)

The following user journeys are currently supported by the Slang Fitness Assistant:

1. Voice Food Log
2. Voice Sugar Log
3. Voice Weight Log

The Action Handler interface has an explicit callback for each of the supported user journeys. Whenever the Assistant detects the user's journey (based on what they spoke), it invokes the callback associated with that user journey. When these callbacks are invoked, the Assistant also passes the parametric data corresponding to the user journey that the Assistant was able to gather. The app is then expected to:

1. Consume the parametric data as needed
2. Optionally launch appropriate UI actions
3. Set appropriate conditions in the Assistant based on the app's internal state
4. Return the AppState that the app transitioned to

Registering for events

The app can register with the Assistant to be notified of all interesting life-cycle events via the setLifeCycleObserver method

const fitnessAssistantLifeCycleObserver = {
  onAssistantInitSuccess: () => {
  },

  onAssistantInitFailure: error => {
  },

  onAssistantInvoked: () => {
  },

  onAssistantClosed: isCancelled => {
  },

  onAssistantLocaleChanged: locale => {
  },

  onUnrecognisedUtterance: utterance => {
  },

  onUtteranceDetected: utterance => {
  },

  onOnboardingSuccess: () => {
  },

  onOnboardingFailure: () => {
  },

  onMicPermissionDenied: () => {
  },

  onMicPermissionGranted: () => {
  },

};

SlangFitnessAssistant.setLifecycleObserver(fitnessAssistantLifeCycleObserver);

As part of the Lifecycle Events API, an observer will be notified of the following events:

1. onAssistantInitSuccess Called as soon as the Assistant has initialized successfully and is ready to serve the app
2. onAssistantInitFailure Called when the Assistant failed to initialize successfully. The reason is passed as a parameter to this callback
3. onAssistantInvoked Called whenever the Assistant has been launched (this can be either as a result of the user clicking on the trigger or the app invoking the Assistant via the startConversation API)
4. onAssistantClosed Called whenever the Assistant has been dismissed. A boolean parameter isCanceled is passed to indicate whether this happened because the user canceled the session or if the Assistant was done with its job
5. onAssistantLocaleChanged Called whenever the user changes the locale of the Assistant
6. onUnrecognisedUtterance Called whenever the Assistant is not able to understand what the user spoke. The utterance that the user spoke is passed as a parameter.
7. onUtteranceDetected Called whenever the Assistant has detected the user uttearnce. The utterance that the user spoke is passed as a parameter.
8. onOnboardingSuccess Called when the Assistant onboarding process has completed successfully.
9. onOnboardingFailure Called when the Assistant onboarding process has failed.
10. onMicPermissionDenied Called when the mic permission for the assistant was denied by the user.
11. onMicPermissionGranted Called when the mic permission for the assistant was granted by the user.

The onFoodLog callback

onFoodLog: (foodInfo, foodUserJourney)

When this callback is invoked, the app is expected to:

1. Consume the details of the foodLog request via the FoodInfo parameter.
2. Fire the app's foodLog request.
3. Finally, return the AppState along with the appropriate Condition corresponding to the state that the app transitioned into

For example, for a given onFoodLog callback invocation, if the food log completes successfully and the app transitions to a screen showing the logged food items, the app would return theAppState as FoodLoggingCompleteAppState along with condition SUCCESS , as shown below:

  onFoodLog: (foodInfo, foodLoggingUserJourney) => {
    // Handle the food log request    
    // ...
    
    return FoodLoggingUserJourney.FoodLoggingCompleteAppState(
        FoodLoggingUserJourney.AppStateCondition.SUCCESS
    )
    
  }

Sample Utterances that could trigger the food log

1. "1 plate idli"
2. "2 plate idli 1 plate vada"
3. "5 pieces of dosa for breakfast"
4. "1 glass of milk for dinner"
5. "1 plate rava dosa today at 11:00 AM"

FoodInfo Parameter

The FoodInfo contains the breakdown of the original food log request. Its structure is as described below:

// When the user searches for something like 
// 1 plate idli and 1 plate dosa and 1 plate sushi for breakfast on 15th of December at 10 AM
// This is how the foodInfo parameter would be populated
{
   "mealType":"BREAKFAST",
   "logDate":"15-12-2021",
   "logTime":"10:00:00",
   "confirmationStatus" : "UNKNOWN"
   "foods":[
      {
         "foodName":"idli",
         "foodQuantity" : {
           "amount":1,
           "unit":"PLATE",
         }
         "isRecognized":true
      },
      {
         "foodName":"dosa",
         "foodQuantity" : {
           "amount":1,
           "unit":"PLATE",
         }
         "isRecognized":true
      },
      {
         "foodName":"sushi",
         "foodQuantity" : {
           "amount":1,
           "unit":"NUMBER",
         }
         "isRecognized":false
      }
   ]
}

NOTE

MealType includes :

1. "BREAKFAST"
2. "LUNCH"
3. "DINNER"
4. "SNACKS"

confirmationStatus includes :

1. "UNKOWN" (To indicate that confirmation has not taken place)
2. "CONFIRMED" (To indicate that its confirmed)
3. "DENIED" (To indicate that its denied)

The isRecognized field is available to verify if the fitness assistant was able correctly recognize the food item. (It's based on the food items that the assistant NLP model has been trained on). If the food item spoken is not a part of the trained set of food items, the assistant performs a small cleanup and gives back the food item but with this flag set as false to indicate that its not actually recognized by assistant NLP model.

Supported AppStates

The following AppStates are supported:

1. FoodLoggingCompleteAppState : To be returned when the app performs the food log. To indicate whether the food log was successful or not, with a greater level of detail, please use the appropriate conditions.

2. UnsupportedAppState : To be returned when the app is not ready to handle search yet. The Assistant will let the user know that food log is not yet supported by the app.

NOTE The Slang Fitness Assistant provides a special WaitingAppState that is common across all UserJourney types for completing asynchronous operations within the callback.

Supported Conditions

1. FoodLoggingCompleteAppState :
   1. SUCCESS : The food logging was successful
   2. FAILURE : The food logging was not successful
   3. MEAL_TYPE_NOT_SPECIFIED : The meal type was not specified
   4. DATE_NOT_SPECIFIED : The meal date was not specified
   5. INVALID_DATE_SPECIFIED : The given log date is not valid.
   6. TIME_NOT_SPECIFIED : The meal time was not specified
   7. FOODS_NOT_SPECIFIED : The meal foods were not specified
   8. FOODS_LIST_INCOMPLETE : The meal foods were incomplete. (NOTE: This condition is used when we would like to prompt the user to specify more items to log)
   9. CONFIRMATION_REQUIRED : The current Food Log journal needs to be confirmed.
   10. VERIFICATION_REQUIRED : The current Food Log journal needs to be verified.
   11. CONFIRMATION_BAILOUT : The voice confirmation needs to bail out and the user needs to perform the confirmation via touch.
   12. FOODS_FOOD_NAME_DISAMBIGUATE : The particular food name is ambigous and we would like the user to disambiguate that item. (NOTE: This condition additionally requires the user to specify the index of the food item in the list food items via the Context)

   FoodLoggingUserJourney.context.setDisambiguationIndex(1);
   //This will disambiguate the food name for the first item in the list of items.
   
   Additionally, the use can also specify the item via touch and the app can inform the SDK via another API as follows,
   FoodLoggingUserJourney.context.setFoodInfoAtIndexInContext(1, {
       "foodName" : "Rava Idli"
   })

The onSugarLog callback

onSugarLog: (sugarInfo, sugarLoggingUserJourney)

When this callback is invoked, the app is expected to:

1. Consume the details of the sugarLog request via the sugarInfo parameter.
2. Fire the app's sugarLog request.
3. Finally, return the AppState along with the appropriate Condition corresponding to the state that the app transitioned into

For example, for a given onSugarLog callback invocation, if the sugar log completes successfully and the app transitions to a screen showing the logged sugar page, the app would return theAppState as SugarLoggingCompleteAppState along with condition SUCCESS , as shown below:

  onSugarLog: (sugarInfo, sugarLoggingUserJourney) => {
    // Handle the sugar log request    
    // ...
    
    return SugarLoggingUserJourney.SugarLoggingCompleteAppState(
        SugarLoggingUserJourney.AppStateCondition.SUCCESS
    )
    
  }

Sample Utterances that could trigger the sugar log

1. "25 mg today morning at 10 AM"
2. "log my sugar level of 100 today post lunch"

SugarInfo Parameter

The ## SugarInfo Parameter contains the breakdown of the original sugar log request. Its structure is as described below:

// When the user searches for something like 
// 25 mg today morning at 10 AM before breakfast
// This is how the foodInfo parameter would be populated
{
   "logDate":"18-10-2022",
   "logTime":"10:00:00",
   "logPeriod":"PRE_BREAKFAST",
   "sugarValue":25,
   "confirmationStatus" : "UNKNOWN"
}

NOTE

LogPeriod includes :

1. "FASTING"
2. "POST_BREAKFAST"
3. "POST_DINNER"
4. "POST_LUNCH"
5. "POST_SNACKS"
6. "POST_EXERCISE"
7. "PRE_BREAKFAST"
8. "PRE_DINNER"
9. "PRE_LUNCH"
10. "PRE_SNACKS"
11. "PRE_EXERCISE"

confirmationStatus includes :

1. "UNKOWN" (To indicate that confirmation has not taken place)
2. "CONFIRMED" (To indicate that its confirmed)
3. "DENIED" (To indicate that its denied)

Supported AppStates

The following AppStates are supported:

1. SugarLoggingCompleteAppState : To be returned when the app performs the sugar log. To indicate whether the sugar log was successful or not, with a greater level of detail, please use the appropriate conditions.

2. UnsupportedAppState : To be returned when the app is not ready to handle search yet. The Assistant will let the user know that the sugar log is not yet supported by the app.

NOTE The Slang Fitness Assistant provides a special WaitingAppState that is common across all UserJourney types for completing asynchronous operations within the callback.

Supported Conditions

1. SugarLoggingCompleteAppState :
   1. SUCCESS : The sugar logging was successful
   2. FAILURE : The sugar logging was not successful
   3. SUGAR_LEVEL_REQUIRED : The sugar level is required
   4. DATE_REQUIRED : The sugar log date is required
   5. INVALID_DATE_SPECIFIED : The given log date is not valid.
   6. TIME_REQUIRED : The log time is required
   7. CONFIRMATION_REQUIRED : The current Sugar Log journal needs to be confirmed.
   8. VERIFICATION_REQUIRED : The current Sugar Log journal needs to be verified.
   9. CONFIRMATION_BAILOUT : The voice confirmation needs to bail out and the user needs to perform the confirmation via touch.

The onWeightLog callback

onWeightLog: (weightInfo, weightLoggingUserJourney)

When this callback is invoked, the app is expected to:

1. Consume the details of the weightLog request via the weightInfo parameter.
2. Fire the app's weightLog request.
3. Finally, return the AppState along with the appropriate Condition corresponding to the state that the app transitioned into

For example, for a given onWeightLog callback invocation, if the weight log completes successfully and the app transitions to a screen showing the logged weight page, the app would return theAppState as WeightLoggingCompleteAppState along with condition SUCCESS , as shown below:

  onWeightLog: (weightInfo, weightLoggingUserJourney) => {
    // Handle the weight log request    
    // ...
    
    return WeightLoggingUserJourney.WeightLoggingCompleteAppState(
        WeightLoggingUserJourney.AppStateCondition.SUCCESS
    )
    
  }

Sample Utterances that could trigger the weight log

1. "25 kg today morning at 10 AM"
2. "log my weight of 100 today"

WeightInfo Parameter

The ## WeightInfo Parameter contains the breakdown of the original weight log request. Its structure is as described below:

// When the user searches for something like 
// 25 kg today morning at 10 AM
// This is how the weightInfo parameter would be populated
{
   "logDate":"18-10-2022",
   "logTime":"10:00:00",
   "weightValue":25,
   "weightUnit":"kg",
   "confirmationStatus" : "UNKNOWN"
}

NOTE

confirmationStatus includes :

1. "UNKOWN" (To indicate that confirmation has not taken place)
2. "CONFIRMED" (To indicate that its confirmed)
3. "DENIED" (To indicate that its denied)

Supported AppStates

The following AppStates are supported:

1. WeightLoggingCompleteAppState : To be returned when the app performs the weight log. To indicate whether the weight log was successful or not, with a greater level of detail, please use the appropriate conditions.

2. UnsupportedAppState : To be returned when the app is not ready to handle search yet. The Assistant will let the user know that the weight log is not yet supported by the app.

NOTE The Slang Fitness Assistant provides a special WaitingAppState that is common across all UserJourney types for completing asynchronous operations within the callback.

Supported Conditions

1. FoodLoggingCompleteAppState :
   1. SUCCESS : The weight logging was successful
   2. FAILURE : The weight logging was not successful
   3. WEIGHT_REQUIRED : The weight is required
   4. DATE_REQUIRED : The weight log date is required
   5. INVALID_DATE_SPECIFIED : The given log date is not valid.
   6. TIME_REQUIRED : The log time is required
   7. CONFIRMATION_REQUIRED : The current weight Log journal needs to be confirmed.
   8. VERIFICATION_REQUIRED : The current weight Log journal needs to be verified.
   9. CONFIRMATION_BAILOUT : The voice confirmation needs to bail out and the user needs to perform the confirmation via touch.

Asynchronous Action Handling

While handling a user journey, the app might not be able to identify the state or the condition of the journey in a synchronous way. For example, the app could open a new activity (or page) which could then fire up a search to the back-end and show the results. So the callback itself would not be aware of the result unless it blocks till the entire activity is loaded. This is not desirable and leads to poor UX. To solve this problem, the Slang Assistant programming model provides ways for the app to asynchronously signal the Assistant about the app state changes and conditions. This can be done by using the "wait" and "notify" semantics. The process is quite straightforward. In order to deal with asynchronous operations within the callback: The app must return the AppState WAITING to inform the Assistant that it's not yet ready to identify the state or the condition of the app. The Assistant goes into a "waiting" state without blocking the UI or other threads and continues to wait for further notification from the app. When the app has completed the asynchronous operation, it can then call notifyAppState to notify the Assistant to proceed further.

Returning WAITING AppState

The app should return WAITING from the user journey callback. This would keep the Assistant in a "processing" (which is what it would have been when it invoked the callback)

onFoodLog: (foodInfo, foodLoggingUserJourney) => {
    // Fire an async food Info handling request  
    // ...
    
    return FoodLoggingUserJourney.WaitingAppState()
    
  }

  onSugarLog: (sugarInfo, sugarLoggingUserJourney) => {
    // Fire an async sugar Info handling request  
    // ...
    
    return SugarLoggingUserJourney.WaitingAppState()
    
  }

  onWeightLog: (weightInfo, weightLoggingUserJourney) => {
    // Fire an async weight Info handling request  
    // ...
    
    return WeightLoggingUserJourney.WaitingAppState()
    
  }

Notifying the Assistant

Once the Assistant is asked to wait, it will continue to remain in the "processing" state until the app notifies it to stop waiting, or the user cancels the operation through the UI. For best results, the app should ensure that every path that returns WAITING has an accompanying call to notify the Assistant.

To notify the Assistant with the AppState, the app needs access to the foodLoggingUserJourney/sugarLoggingUserJourney/weightLoggingUserJourney object that was passed to the callback. (The App can store the foodLoggingUserJourney/sugarLoggingUserJourney/weightLoggingUserJourney object obtained in the callback to state and then use it once it needs to accessed later)

foodLoggingUserJourney.notifyAppState(
    FoodLoggingUserJourney.FoodLoggingCompleteAppState(
        FoodLoggingUserJourney.AppStateCondition.SUCCESS
    )
);

sugarLoggingUserJourney.notifyAppState(
    SugarLoggingUserJourney.SugarLoggingCompleteAppState(
        SugarLoggingUserJourney.AppStateCondition.SUCCESS
    )
);

weightLoggingUserJourney.notifyAppState(
    WeightLoggingUserJourney.WeightLoggingCompleteAppState(
        WeightLoggingUserJourney.AppStateCondition.SUCCESS
    )
);