mmir-lib

Source Code for the mmir (Mobile Multimodal Interaction and Relay)* library

The mmir framework provides means for creating minimal (client-based) dialog systems for multimodal interactions:

• state-based interaction handling using SCXML (e.g. for touch/click, speech, gesture interactions)
• support and plugins for several speech input (Automatic Speech Recognition, ASR) engines
• support and plugins for several speech output/synthesis (Text To Speech, TTS) engines
• support for client- or sever-based NLU processing
• built-in support for grammars (similar to BNF grammars)

Overview:

• Install
• API Documentation
• Examples
• Pre-Building Grammars, State Machines etc.
• Include in Web Page
• Node.js
  • Node.js and WebWorker
• Additional Notes
• Used Libraries
• License

Usage

Install

• clone repository / download as ZIP file and include /lib directory in your assets (see below for how to include in JavaScript code)
• with npm / package.json
  • install via npm registry:

    npm install mmir-lib

  • install master branch via GitHub repository:

    npm install git+https://github.com/mmig/mmir-lib.git

• see mmir-tooling and mmir-wepack for more options

API Documentation

The API documentation is available at /docs (md format) or in HTML format (more details are available in jsdoc-generated API documentation, and further details at github.com/mmig/mmir).

Examples

For examples, see the mmir-starter-kit or mmir-starter-ionic.

Pre-Building Grammars, State Machines etc.

• Prerequesites: Node.js

Some resources (e.g. grammars, state machines) can be built/compiled before they are used in the application at runtime.
Note that most of these resources can be compiled during runtime too (as in case of grammars and state machines).

Compiling/building these resources is supported e.g. via the mmir-tooling project.

In addition, the mmir-wepack integration provides special support for including mmir in webpack-based/-built applications.

Include in Web Page

The following shows an example of including mmir-lib in a web page "as-is", i.e. without any build-system (like webpack).

mmir-lib includes requirejs for (async) lazy-loading its sub-modules.

• include /lib as directory /mmirf in your web resources directory, e.g. /www/mmirf
  (e.g. use mmirinstall script of mmir-tooling: mmirinstall www/mmirf)

• load/include mmir in HTML page

  <!-- OPTIONAL helper script: auto-detect Cordova-environment and load its library if necessary: -->
  <script type="text/javascript" src="mmirf/tools/initCordova.js"></script>
  
  <!-- load the framework's core/base object -->
  <script type="text/javascript" src="mmirf/core.js"></script>
  
  <!-- OPTIONAL: configure mmir framework before it starts loading/initializing
                  using some custom script (in this example at appjs/preinit.js)
                  for more details see the documentation at github.com/mmig/mmir
  -->
  <script type="text/javascript" src="appjs/preinit.js"></script>
  
  <!-- load mmir library -->
  <script type="text/javascript" src="mmirf/vendor/libs/require.min.js" data-main="mmirf/mainConfig" ></script>

• use mmir in JavaScript code (after it's been initialized)

  mmir.ready(function(){
    ...
  })

Node.js

install via npm

npm instal mmir-lib

initialize and include in JavaScript code:

var mmirLib = require('mmir-lib');

//optional: run some configuration before starting to initialize mmir
var preInitFunc = function(mmir){
  //... some custom configuration for mmir before loading the library
};

//init should only be called once:
var mmir = mmirLib.init(preInitFunc);

mmir.ready(function(){
  ...
});

Node.js and WebWorker

For some functionality (e.g. async-compiling grammars, state-manager event processing), mmir uses WebWorkers. While most of the these functionalities do have fallback implementations, they may not always provide the same functionality, and run less efficiently.

mmir can use the following WebWorker implementations as drop-ins for HTML5 WebWorkers when running in node:

• built-in node module worker_threads (recommended):

  • since version 10.5.0 available as experimental feature:
    use/enable by running node with command-line argument --experimental-worker, e.g.

    # when using node directly
    node --experimental-worker ...
    # when using npm
    npm --node-options --experimental-worker ...
    # set as node env variable (*nix)
    export NODE_OPTIONS=$NODE_OPTIONS --experimental-worker
    # set as node env variable (Windows)
    set NODE_OPTIONS=%NODE_OPTIONS% --experimental-worker
    # add entry in npm configuration file .npmrc:
    #node-options = --experimental-worker

  • since node version 11.x worker_threads is enabled by default

• alternative module webworker-threads with version >= 0.8.x

  • NOTE not everything may work using this package (if possible use worker_threads)
  • worker_threads at npm: current version is
  • worker_threads on github: current version is

Additional Notes

More details, further documentation etc. are available on the mmir project page.

See also the tools project for scripts, resources etc. for compiling and generating resources (e.g. for compiling JSON grammar files from the application's config/languages/[language code]/ directories into JavaScript files).

NOTE: Integration with / loading of Cordova is designed to work with the build process of Cordova 5 or later (see mmir-tooling and cordova example project for integration / tooling of the mmir framework in combination with Cordova 5 and later versions).

Used Libraries

(see contents of /vendor/libs/)

• RequireJS 2.3.7
  (BSD or MIT; Copyright jQuery Foundation and other contributors)
• SCION v1, @scion-scxml/core v2.6.22 (custom build)
  (LGPLv3, Apache License v2.0; Copyright 2018 Jacobean Research and Development, LLC)
• JS/CC 0.30
  (BSD; Copyright © 2007-2016 by Phorward Software Technologies; Jan Max Meyer; Brobston Development, Inc.; and other contributors)
• PEG.js 0.10.0
  (MIT; Copyright (c) 2010-2016 David Majda, Copyright (c) 2017+ Futago-za Ryuu)
• Jison 0.4.18 (modified build)
  (MIT; Copyright (c) 2009-2014 Zachary Carter)
• ANTLR 3
  (BSD; Copyright (c) 2013 Terence Parr)
• crypto-js MD5 3.1.9-1 (custom build)
  (MIT; Copyright (c) 2009-2013 Jeff Mott, Copyright (c) 2013-2016 Evan Vosberg)
• stacktrace-js 2.0.2
  (MIT; Copyright (c) 2017 Eric Wendelin and other contributors)

License

If not stated otherwise, all files and resources are provided under the MIT license

*: since version 6 the last letter of the acronym MMIR has been reasigned from Rendering to Relay in order to account for the fact, that (graphical) rendering is not a major focus of the the framework anymore.