Runs word-error-rate evaluation on Watson STT Model

Install

npm install -g watson-stt-experiment

Usage

CLI:

> watson-stt-experiment

Marcão WER Evaluation Script

  Runs WER evaluation on Watson Speech-to-text Model. 

Options

  -h, --help                             Print usage instructions.                         
  -u, --url string                       Watson STT base URL.                              
  -a, --apikey string                    Watson STT API Key.                               
  -f, --filePath string                  CSV file with columns [audio, transcript] 
  -m, --model string                     Watson STT base model ID.                         
  -c, --customizationId string           Language customization ID                         
  -d, --acousticCustomizationId string   Acoustic customization ID                         
  -v, --version string                   Watson STT API version. Default: 2020-07-01       
  -o, --output string                    Output file. Default: results.json                

Output

  WER Evaluation results in JSON format 

As a module:

const STT = require('watson-stt-experiment')
const stt = new STT({ 
  version: '2020-07-01', 
  apikey: 'YOUR_WATSON_STT_API_KEY', 
  url: 'YOUR_WATSON_STT_SERVICE_URL'
})

let results = await stt.runExperiment({  
  groundTruth: [
    {
      audio: "some_dir/audio_1.mp3",
      transcript: "How to change my password"
    },
    {
      audio: "some_dir/audio_2.mp3",
      transcript: "How do I change my password"
    }
  ], 
  model: 'en-US_BroadbandModel'
})

Sample results

{
    "total_words": 11,
    "word_error_rate": 0.90909091,
    "sentence_error_rate": 0.5,
    "transcriptions": [
        {
            "file": "some_dir/audio_1.mp3",
            "text": "How to change my password",
            "prediction": "How to change my password",
            "word_error_rate": 0,
            "changes": []
        },
        {
            "file": "some_dir/audio_2.mp3",
            "text": "How do I change my password",
            "prediction": "How I change my password",
            "word_error_rate": 0.16666666666666666,
            "changes": [
                {
                    "type": "deletion",
                    "phrase": "do"
                }
            ]
        }
    ]
}

Supported change types are currently:

• addition
• deletion
• substitution

Run tests

npm run test

Author

👤 Marco Cardoso

• Github: @MarcoABCardoso
• LinkedIn: @marco-cardoso

Show your support

Give a ⭐️ if this project helped you!