Lexer

A small library for parse code in tokens

Table of Contents

• Installation
• Usage
• Docs
• License

Installation

Install Globally

  npm i -g @albertocruzluis/lexer

Install Locally

  npm i @albertocruzluis/lexer --save

Usage

Import of the lexer module

import buildLexer from '@albertocruzluis/lexer'

we create the tokens that our parser will be able to recognise

const SPACE = /(?<SPACE>\s+|\/\/.*)/;
const RESERVEDWORD = /(?<RESERVEDWORD>\b(const|let)\b)/;
const ID = /(?<ID>\b([a-z_]\w*))\b/;
const STRING = /(?<STRING>"([^\\"]|\\.")*")/;
const OP = /(?<OP>[+*\/=-])/;

const tokens = [
  ['SPACE', SPACE, true], ['RESERVEDWORD', RESERVEDWORD], ['ID', ID], 
  ['STRING', STRING], ['OP', OP] 
];

const lexer = buildLexer(tokens);

Example generate tokens of a code

const str = 'const varName = "value"';
const result = lexer(str);
console.log(result);
/* 
  [
    { type: 'RESERVEDWORD', value: 'const' },
    { type: 'ID', value: 'varName' },
    { type: 'OP', value: '=' },
    { type: 'STRING', value: '"value"' }
  ]
*/

Docs

See documentation for more details.

License

MIT