npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

web-tree-sitter

v0.24.4

Published

Tree-sitter bindings for the web

Downloads

2,328,214

Readme

Web Tree-sitter

npmjs.com badge

WebAssembly bindings to the Tree-sitter parsing library.

Setup

You can download the tree-sitter.js and tree-sitter.wasm files from the latest GitHub release and load them using a standalone script:

<script src="/the/path/to/tree-sitter.js"></script>

<script>
  const Parser = window.TreeSitter;
  Parser.init().then(() => { /* the library is ready */ });
</script>

You can also install the web-tree-sitter module from NPM and load it using a system like Webpack:

const Parser = require('web-tree-sitter');
Parser.init().then(() => { /* the library is ready */ });

You can use this module with deno:

import Parser from "npm:web-tree-sitter";
await Parser.init();
// the library is ready

Basic Usage

First, create a parser:

const parser = new Parser;

Then assign a language to the parser. Tree-sitter languages are packaged as individual .wasm files (more on this below):

const JavaScript = await Parser.Language.load('/path/to/tree-sitter-javascript.wasm');
parser.setLanguage(JavaScript);

Now you can parse source code:

const sourceCode = 'let x = 1; console.log(x);';
const tree = parser.parse(sourceCode);

and inspect the syntax tree.

console.log(tree.rootNode.toString());

// (program
//   (lexical_declaration
//     (variable_declarator (identifier) (number)))
//   (expression_statement
//     (call_expression
//       (member_expression (identifier) (property_identifier))
//       (arguments (identifier)))))

const callExpression = tree.rootNode.child(1).firstChild;
console.log(callExpression);

// { type: 'call_expression',
//   startPosition: {row: 0, column: 16},
//   endPosition: {row: 0, column: 30},
//   startIndex: 0,
//   endIndex: 30 }

Editing

If your source code changes, you can update the syntax tree. This will take less time than the first parse.

// Replace 'let' with 'const'
const newSourceCode = 'const x = 1; console.log(x);';

tree.edit({
  startIndex: 0,
  oldEndIndex: 3,
  newEndIndex: 5,
  startPosition: {row: 0, column: 0},
  oldEndPosition: {row: 0, column: 3},
  newEndPosition: {row: 0, column: 5},
});

const newTree = parser.parse(newSourceCode, tree);

Parsing Text From a Custom Data Structure

If your text is stored in a data structure other than a single string, you can parse it by supplying a callback to parse instead of a string:

const sourceLines = [
  'let x = 1;',
  'console.log(x);'
];

const tree = parser.parse((index, position) => {
  let line = sourceLines[position.row];
  if (line) return line.slice(position.column);
});

Generate .wasm language files

The following example shows how to generate .wasm file for tree-sitter JavaScript grammar.

IMPORTANT: emscripten, docker, or podman need to be installed.

First install tree-sitter-cli and the tree-sitter language for which to generate .wasm (tree-sitter-javascript in this example):

npm install --save-dev tree-sitter-cli tree-sitter-javascript

Then just use tree-sitter cli tool to generate the .wasm.

npx tree-sitter build --wasm node_modules/tree-sitter-javascript

If everything is fine, file tree-sitter-javascript.wasm should be generated in current directory.

Running .wasm in Node.js

Notice that executing .wasm files in node.js is considerably slower than running node.js bindings. However could be useful for testing purposes:

const Parser = require('web-tree-sitter');

(async () => {
  await Parser.init();
  const parser = new Parser();
  const Lang = await Parser.Language.load('tree-sitter-javascript.wasm');
  parser.setLanguage(Lang);
  const tree = parser.parse('let x = 1;');
  console.log(tree.rootNode.toString());
})();

Running .wasm in browser

web-tree-sitter can run in the browser, but there are some common pitfalls.

Loading the .wasm file

web-tree-sitter needs to load the tree-sitter.wasm file. By default, it assumes that this file is available in the same path as the JavaScript code. Therefore, if the code is being served from http://localhost:3000/bundle.js, then the wasm file should be at http://localhost:3000/tree-sitter.wasm.

For server side frameworks like NextJS, this can be tricky as pages are often served from a path such as http://localhost:3000/_next/static/chunks/pages/index.js. The loader will therefore look for the wasm file at http://localhost:3000/_next/static/chunks/pages/tree-sitter.wasm. The solution is to pass a locateFile function in the moduleOptions argument to Parser.init():

await Parser.init({
  locateFile(scriptName: string, scriptDirectory: string) {
    return scriptName;
  },
});

locateFile takes in two parameters, scriptName, i.e. the wasm file name, and scriptDirectory, i.e. the directory where the loader expects the script to be. It returns the path where the loader will look for the wasm file. In the NextJS case, we want to return just the scriptName so that the loader will look at http://localhost:3000/tree-sitter.wasm and not http://localhost:3000/_next/static/chunks/pages/tree-sitter.wasm.

Can't resolve 'fs' in 'node_modules/web-tree-sitter'

Most bundlers will notice that the tree-sitter.js file is attempting to import fs, i.e. node's file system library. Since this doesn't exist in the browser, the bundlers will get confused. For webpack you can fix this by adding the following to your webpack config:

{
  resolve: {
    fallback: {
      fs: false
    }
  }
}