MindScript Tokenizer

This is a simple mindscript tokenizer to make your chatbot better understand user questions.

How to Use

1. Install the dependencies:

npm install @mindlearn/tokenizer

2. Use the tokenizer in your code:

const tokenizer = require("@mindlearn/tokenizer");

const word = "Hello World, here is a basic mindscript tokenizer example.";
console.log(tokenizer(word));