wordmap-lexer
v0.3.6
Published
A lexer used in wordMAP
Downloads
468
Maintainers
Keywords
Readme
wordMAP-lexer
A lexical tool for wordMAP
The lexer provides tools generating tokens which can be fed into WordMAP.
Install
npm i wordmap-lexer
Usage
require Lexer, {Token} from 'wordmap-lexer';
const tokens : Token[] = Lexer.tokenize("hello, world");
// produces tokens "hello" and "world".
const puct_tokens : Token[] = Lexer.tokenize("hello, world", {punctuation: true});
// produces tokens "hello", ",", and "world".