token-flow

token-flow is an exerimental natural language tokenizer intended for domains with a fixed vocabulary of entities and a small number of intents. Uses might include purchasing items from a catalog, ordering food from a restaurant or organizing your song collection.

The first stage of development is a set of tokenizers that detect entities, intents, and quantities. As an example, consider the following utterance, which would typically come from a speech-to-text system:

I'd like a black sedan with alloy wheels skip the extended warantty and a red convertable jacked with open headers

In this example, the utterance has no commas, since they were not provided by the speech-to-text process.

Using token-flow, this text might be tokenized as

[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:BLACK_FOUR_DOOR_SEDAN,20]
[QUANTITY:1] [ENTITY:ALLOY_RIMS,1000] [QUANTITY:0] [ENTITY:EXTENDED_WARRANTY,1800]
[INTENT:CONJUNCTION] [QUANTITY:1] [ENTITY:RED_TWO_DOOR_CONVERTIBLE_SEDAN,1]
[ENTITY:LIFT_KIT,1200] [QUANTITY:1] [ENTITY:OPEN_HEADERS,1203]

After tokenization, a parser might be able to group the tokens into a tree that reflects the speaker's intent:

[INTENT:ADD_TO_ORDER]
    [QUANTITY:1] [ENTITY:BLACK_FOUR_DOOR_SEDAN,20]          // Black sedan
        [QUANTITY:1] [ENTITY:ALLOY_RIMS,1000]               //   Add alloy rims
        [QUANTITY:0] [ENTITY:EXTENDED_WARRANTY,1800]        //   Remove warranty
    [QUANTITY:1] [ENTITY:RED_TWO_DOOR_CONVERTIBLE_SEDAN,1]  // Red convertable
        [ENTITY:LIFT_KIT,1200]                              //   Add lift kit
        [QUANTITY:1] [ENTITY:OPEN_HEADERS,1203]             //   Make it loud

Try It Out

token-flow is currently in the earliest stages of development, so documentation is sparse or nonexistant, and the code stability is uneven.

If you are interested in taking a look, you can clone the repo on GitHub or install token-flow with npm.

npm install token-flow

As of commit 87407400, there are a number of working samples, based on a ficticious auto dealership.

Note that the samples are not included in the token-flow npm package. To use them, you must clone the repo from GitHub.

You can find the definition files for the catalog, intents, attributes, and quantifiers at

• samples/data/cars/catalog.yaml
• samples/data/intents.yaml
• samples/data/attributes.yaml
• samples/data/quantifiers.yaml

Relevance Test Sample

This sample runs a suite of test utterances through the tokenization pipeline. The test utterances can be found at samples/data/cars/tests.yaml.

If you've cloned the repo, you can build and run the sample as follows:

npm install
npm run compile
node build/samples/relevance_demo_cars.js

The output is the sequence of tokens extracted for each test utterance:

% node build/samples/relevance_demo_cars.js

14 items contributed 145 aliases.
5 items contributed 29 aliases.
21 items contributed 30 aliases.
78 items contributed 212 aliases.

All tests passed.

0 general - PASSED
   input "I'd like a black sedan with alloy wheels skip the extended warantty and a red convertable jacked with open headers"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:BLACK_FOUR_DOOR_SEDAN,20] [QUANTITY:1] [ENTITY:ALLOY_RIMS,1000] [QUANTITY:0] [ENTITY:EXTENDED_WARRANTY,1800] [INTENT:CONJUNCTION] [QUANTITY:1] [ENTITY:RED_TWO_DOOR_CONVERTIBLE_SEDAN,1] [ENTITY:LIFT_KIT,1200] [QUANTITY:1] [ENTITY:OPEN_HEADERS,1203]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:BLACK_FOUR_DOOR_SEDAN,20] [QUANTITY:1] [ENTITY:ALLOY_RIMS,1000] [QUANTITY:0] [ENTITY:EXTENDED_WARRANTY,1800] [INTENT:CONJUNCTION] [QUANTITY:1] [ENTITY:RED_TWO_DOOR_CONVERTIBLE_SEDAN,1] [ENTITY:LIFT_KIT,1200] [QUANTITY:1] [ENTITY:OPEN_HEADERS,1203]"

1 general - PASSED
   input "convertible with tinted windows and fuzzy dice"
  output "[ENTITY:RED_TWO_DOOR_CONVERTIBLE_SEDAN,1] [QUANTITY:1] [ENTITY:TINTED_WINDOWS,1205] [INTENT:CONJUNCTION] [ENTITY:FUZZY_DICE,1600]"
expected "[ENTITY:RED_TWO_DOOR_CONVERTIBLE_SEDAN,1] [QUANTITY:1] [ENTITY:TINTED_WINDOWS,1205] [INTENT:CONJUNCTION] [ENTITY:FUZZY_DICE,1600]"

2 general - PASSED
   input "I'd like a four door sedan with moon roof trailer hitch and tinted windows"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:SILVER_FOUR_DOOR_SEDAN,12] [QUANTITY:1] [ENTITY:MOON_ROOF,1302] [ENTITY:TOW_PACKAGE,1303] [INTENT:CONJUNCTION] [ENTITY:TINTED_WINDOWS,1205]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:SILVER_FOUR_DOOR_SEDAN,12] [QUANTITY:1] [ENTITY:MOON_ROOF,1302] [ENTITY:TOW_PACKAGE,1303] [INTENT:CONJUNCTION] [ENTITY:TINTED_WINDOWS,1205]"

3 general - PASSED
   input "Give me the monster truck jacked with knobbies glass packs open headers and an air freshener"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:GOLD_MONSTER_TRUCK,30] [ENTITY:LIFT_KIT,1200] [QUANTITY:1] [ENTITY:KNOBBY_TIRES,1004] [ENTITY:GLASS_PACKS,1204] [ENTITY:OPEN_HEADERS,1203] [INTENT:CONJUNCTION] [QUANTITY:1] [ENTITY:PINE_SCENTED_AIR_FRESHENER,1601]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:GOLD_MONSTER_TRUCK,30] [ENTITY:LIFT_KIT,1200] [QUANTITY:1] [ENTITY:KNOBBY_TIRES,1004] [ENTITY:GLASS_PACKS,1204] [ENTITY:OPEN_HEADERS,1203] [INTENT:CONJUNCTION] [QUANTITY:1] [ENTITY:PINE_SCENTED_AIR_FRESHENER,1601]"

4 general - PASSED
   input "I want a blue convertible four on the floor no undercoat no warranty"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:BLUE_TWO_DOOR_CONVERTIBLE_SEDAN,5] [ENTITY:FOUR_SPEED_MANUAL_TRANSMISSIONS,1300] [QUANTITY:0] [ENTITY:UNDER_COAT,1304] [QUANTITY:0] [ENTITY:EXTENDED_WARRANTY,1800]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:BLUE_TWO_DOOR_CONVERTIBLE_SEDAN,5] [ENTITY:FOUR_SPEED_MANUAL_TRANSMISSIONS,1300] [QUANTITY:0] [ENTITY:UNDER_COAT,1304] [QUANTITY:0] [ENTITY:EXTENDED_WARRANTY,1800]"

5 general - PASSED
   input "Can I have a silver four door sedan with leather interior and a dump truck"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:SILVER_FOUR_DOOR_SEDAN,12] [QUANTITY:1] [ENTITY:LEATHER_INTERIOR,1700] [INTENT:CONJUNCTION] [QUANTITY:1] [ENTITY:GREY_DUMP_TRUCK,33]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:SILVER_FOUR_DOOR_SEDAN,12] [QUANTITY:1] [ENTITY:LEATHER_INTERIOR,1700] [INTENT:CONJUNCTION] [QUANTITY:1] [ENTITY:GREY_DUMP_TRUCK,33]"

6 general - PASSED
   input "I'll take the school bus actually make that the dump truck"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:YELLOW_SCHOOL_BUS,32] [INTENT:CANCEL_LAST_ITEM] [INTENT:RESTATE] [QUANTITY:1] [ENTITY:GREY_DUMP_TRUCK,33]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:1] [ENTITY:YELLOW_SCHOOL_BUS,32] [INTENT:CANCEL_LAST_ITEM] [INTENT:RESTATE] [QUANTITY:1] [ENTITY:GREY_DUMP_TRUCK,33]"

7 general - PASSED
   input "I'd like two blue sedans one of them jacked with slicks and the other a low rider with moon roof"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:2] [ENTITY:BLUE_TWO_DOOR_SEDAN,6] [QUANTITY:1] [INTENT:PREPOSITIONS] [ENTITY:LIFT_KIT,1200] [QUANTITY:1] [ENTITY:RACING_SLICKS,1006] [INTENT:CONJUNCTION] [QUANTITY:1] [INTENT:PREPOSITIONS] [QUANTITY:1] [ENTITY:LOW_RIDER_KIT,1201] [QUANTITY:1] [ENTITY:MOON_ROOF,1302]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:2] [ENTITY:BLUE_TWO_DOOR_SEDAN,6] [QUANTITY:1] [INTENT:PREPOSITIONS] [ENTITY:LIFT_KIT,1200] [QUANTITY:1] [ENTITY:RACING_SLICKS,1006] [INTENT:CONJUNCTION] [QUANTITY:1] [INTENT:PREPOSITIONS] [QUANTITY:1] [ENTITY:LOW_RIDER_KIT,1201] [QUANTITY:1] [ENTITY:MOON_ROOF,1302]"

8 general - PASSED
   input "Get me two air fresheners one strawberry and the other new car actually make that vanilla"
  output "[INTENT:ADD_TO_ORDER] [QUANTITY:2] [ENTITY:PINE_SCENTED_AIR_FRESHENER,1601] [QUANTITY:1] [ATTRIBUTE:AIR_FRESHENER(STRAWBERRY),302] [INTENT:CONJUNCTION] [QUANTITY:1] [INTENT:PREPOSITIONS] [ATTRIBUTE:AIR_FRESHENER(NEW_CAR),305] [INTENT:CANCEL_LAST_ITEM] [INTENT:RESTATE] [ATTRIBUTE:AIR_FRESHENER(VANILLA),304]"
expected "[INTENT:ADD_TO_ORDER] [QUANTITY:2] [ENTITY:PINE_SCENTED_AIR_FRESHENER,1601] [QUANTITY:1] [ATTRIBUTE:AIR_FRESHENER(STRAWBERRY),302] [INTENT:CONJUNCTION] [QUANTITY:1] [INTENT:PREPOSITIONS] [ATTRIBUTE:AIR_FRESHENER(NEW_CAR),305] [INTENT:CANCEL_LAST_ITEM] [INTENT:RESTATE] [ATTRIBUTE:AIR_FRESHENER(VANILLA),304]"

Suites:
  general: 9/9

Priorities:
  1: 9/9

Overall: 9/9

REPL Sample

This sample provides a Read-Eval-Print-Loop that runs the tokenizer on each line entered.

If you've cloned the repo, you can build and run the sample as follows:

npm run compile
node build/samples/repl_demo.js

% node build/samples/repl_demo.js

Welcome to the token-flow REPL.
Type your order below.
A blank line exits.

14 items contributed 145 aliases.
5 items contributed 23 aliases.
16 items contributed 31 aliases.
78 items contributed 212 aliases.

% I'd like a 1/2 ton truck with the tow package

********************************************************
PLEASE NOTE: your input has been modified to be more
like the output of a speech-to-text system.
your input: "I'd like a 1/2 ton truck with the tow package"
modified:   "I'd like a half ton truck with the tow package"
********************************************************
INTENT: ADD_TO_ORDER: "I'd like"
QUANTITY: 1: "a"
ENTITY: BLACK_HALF_TON_PICKUP_TRUCK(34): "half ton truck"
QUANTITY: 1: "with the"
ENTITY: TOW_PACKAGE(1303): "tow package"

% Can I get a convertable four on the floor with moon roof and air freshener actually make that an automatic

INTENT: ADD_TO_ORDER: "Can I get"
QUANTITY: 1: "a"
ENTITY: RED_TWO_DOOR_CONVERTIBLE_SEDAN(1): "convertable"
ENTITY: FOUR_SPEED_MANUAL_TRANSMISSIONS(1300): "four on the floor"
QUANTITY: 1: "with"
ENTITY: MOON_ROOF(1302): "moon roof"
INTENT: CONJUNCTION: "and"
ENTITY: PINE_SCENTED_AIR_FRESHENER(1601): "air freshener"
INTENT: CANCEL_LAST_ITEM: "actually"
INTENT: RESTATE: "make that"
QUANTITY: 1: "an"
ENTITY: SIX_SPEED_AUTOMATIC_TRANSMISSIONS(1301): "automatic"

%
bye

Stemmer Confusion Matrix Sample

In some cases, the stemmer can stem words with different meanings to the same term. One can check for these problems in the catalog.json, quantifiers.json, and intents.json files by generating a stemmer confusion matrix.

node build\samples\stemmer_confusion_demo.js

14 items contributed 145 aliases.
5 items contributed 29 aliases.
23 items contributed 36 aliases.
83 items contributed 223 aliases.
"wall": [wall,walls]
"knobbi": [knobby,knobbies]
"inject": [injection,injected]
"lift": [lift,lifted]
"glass": [glass,glasses]
"seat": [seats,seat]
"custom": [customer,custom]
"organ": [organizer,organ,organic]
"that": [that,that's]
"thank": [thank,thanks]

In the example above, we see that the words "organizer" and "organ" are treated as the same term, as are "customer" and "custom". This causes the phrase "custom organ" to match "customer organizer folder" instead of "custom organ horn".

% I want a red convertible with a custom organ

INTENT: ADD_TO_ORDER: "I want"
QUANTITY: 1: "a"
ENTITY: RED_TWO_DOOR_CONVERTIBLE_SEDAN(1): "red convertible"
QUANTITY: 1: "with a"
ENTITY: CUSTOMER_ORGANIZER_FOLDER(2000): "custom organ"

% I want a red convertible with a custom organ horn

INTENT: ADD_TO_ORDER: "I want"
QUANTITY: 1: "a"
ENTITY: RED_TWO_DOOR_CONVERTIBLE_SEDAN(1): "red convertible"
QUANTITY: 1: "with a"
ENTITY: CUSTOM_ORGAN_HORN(2001): "custom organ horn"

One can address this problem with a different stemmer or lemmatizer. One simple work-around is to wrap the default stemmer in a function that has special handling for certain words like "organ" and "organizer":

function hackedStemmer(term: string): string {
    const lowercase = term.toLowerCase();
    if (lowercase === 'organ' || lowercase === 'organizer') {
        return lowercase;
    }
    return Tokenizer.defaultStemTerm(lowercase);
}

Tokenizer Design Notes