tokenizer-next
v3.0.1
Published
iterator based tokenizer for writing parsers
Downloads
50
Maintainers
Readme
iterator based tokenizer for writing parsers
API
To capture, RegExps must use a named group.
const tokenize = createTokenizer(
/(?<ident>[a-z]+)/, // named groups determine token `group`
/(?<number>[0-9]+)/
)
// using next()
const next = tokenize('hello 123')
console.log(next()) // => {group: 'ident', value: 'hello', index: 0}
console.log(next()) // => {group: 'number', value: '123', index: 6}
console.log(next()) // => undefined
// using for of
for (const token of tokenize('hello 123')) {
console.log(token)
// => {group: 'ident', value: 'hello', index: 0}
// => {group: 'number', value: '123', index: 6}
}
// using spread
const tokens = [...tokenize('hello 123')]
console.log(tokens)
// => [
// {group: 'ident', value: 'hello', index: 0},
// {group: 'number', value: '123', index: 6}
// ]
To capture, RegExps must use a named group.
const tokenize = createTokenizer(
/(?<ident>[a-z]+)/, // named groups determine token `group`
/(?<number>[0-9]+)/
)
Credits
- match-to-token by stagas – transform a RegExp named group match to a more useful object
Contributing
All contributions are welcome!
License
MIT © 2022 stagas