hx-tokenizer
v1.0.0
Published
Split text into tokens.
Downloads
3
Maintainers
Readme
tokenizer
Split text into tokens.
Install
$ npm install --save hx-tokenizer
Usage
const tokenizer = require("hx-tokenizer");
const tokens = tokenizer.tokenize("Lorem ipsum, something.");
// tokens == ["Lorem", "ipsum", ",", "something", "."]
const text = tokenizer.join(tokens);
// text == "Lorem ipsum, something."
API
tokenizer.tokenize(text)
tokenizer.join(tokens)
License
MIT © Hexacta