npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

vscode-textmate-languageservice

v4.0.0

Published

Textmate token-based language service for Visual Studio Code.

Downloads

149

Readme

vscode-textmate-languageservice

🎉 This package has been adopted by the vsce-toolroom GitHub collective.

This package is in LTS mode & the Textmate technology is superseded by the tree-sitter symbolic-expression parser technology, as used in vscode-anycode.

Language service providers & APIs driven entirely by your Textmate grammar and one configuration file.

To use the API methods and tokenization / outline services, you only need a Textmate grammar. This can be from your extension or one of VS Code's built-in languages.

In order to properly generate full-blown language providers from this module, the Textmate grammar must also include the following features:

  • meta declaration scopes for block level declarations
  • variable assignment scopes differentiated between multiple and single
  • granular keyword control tokens with begin and end scopes

Installation

npm install vscode-textmate-languageservice

Browser support:

  • This package supports Webpack and ESBuild.
  • If you use a bundler, you need to set crypto as a external (commonjs crypto one in webpack). This allows the library to avoid polyfilling the node:crypto module.

Advisory:

This package is stable with browser compatibility (1.1.0). But I recommend you watch out for tree-sitter native integration into vscode (issue). Maintainable & with faster retokenization, it is a Holy Grail ...

Whereas this package depends on a well-written Textmate grammar and is a band aid of sorts.

If there is native vscode support for the language, find a Tree-sitter syntax online then suggest it in an Anycode issue. Otherwise, please open an issue on the community-maintained Treesitter syntax highlighter extension and someone might deal with it.

Setup

  • Language contribution and grammar contribution defined via contributes in the extension manifest (or textmate-languageservice-contributes).
  • Your grammar is bundled in the extension source code and is consumable by vscode-textmate (which can load PList XML, JSON or YAML grammars).
  • A configuration file is available in the extension, defaulting to ./textmate-configuration.json. You can also use textmate-languageservices property of package.json to map language ID to relative path.

Example language extension manifest - ./package.json:

{
	"name": "lua",
	"displayName": "Textmate language service for Lua",
	"description": "Lua enhanced support for Visual Studio Code",
	"version": "0.0.1",
	"publisher": "",
	"license": "",
	"engines": {
		"vscode": "^1.51.1"
	},
	"categories": [
		"Programming Languages"
	],
	"contributes": {
		"languages": [{
			"id": "lua",
			"aliases": ["Lua"],
			"extensions": [".lua", ".moon", ".luau"],
			"configuration": "./language-configuration.json"
		}],
		"grammars": [{
			"language": "lua",
			"scopeName": "source.lua",
			"path": "./syntaxes/Lua.tmLanguage.json"
		}]
	}
}

Configuration

Create a JSON file named textmate-configuration.json in the extension directory. The file accepts comments and trailing commas.

If you only want to use the document and/or tokenization services, you can skip creating the file!

Textmate configuration fields:

  • assignment - optional (object) Collection of Textmate scope selectors for variable assignment scopes when including variable symbols: Properties:
    • separator: Token to separate multiple assignments (string)
    • single: Token to match single variable assignment. (string)
    • multiple: Token to match multiple variable assignment. (string)
  • declarations - optional (array) List of Textmate scope selectors for declaration token scopes.
  • dedentation - optional (array) List of Textmate tokens for dedented code block declarations (e.g. ELSE, ELSEIF). Tokens still need to be listed in indentation with the decrementing value -1.
  • exclude (string) VS Code glob pattern for files to exclude from workspace symbol search.
  • indentation - optional (object) Indentation level offset for Textmate token types (used to implement folding).
  • punctuation - optional (object) Collection of punctuation tokens with a significant effect on syntax providers. Properties:
    • continuation: Token scope selector for line continuation (to use in region matching). (string)
  • markers - optional (object) Stringified regular expression patterns for folding region comments.
    • start: Escaped regular expression for start region marker. (string)
    • end: Escaped regular expression for end region marker. (string) Properties:
  • symbols - optional (object) Map of document symbol tokens to their symbol kind (vscode.SymbolKind value).

Configuration examples

Template for textmate-configuration.json file:

{
  "assignment": {
    "single": "",
    "multiple": "",
    "separator": ""
  },
  "declarations": [],
  "dedentation": [
    "keyword.control.elseif.custom",
    "keyword.control.else.custom"
  ],
  "exclude": "{.modules,.includes}/**",
  "indentation": {
    "punctuation.definition.comment.begin.custom": 1,
    "punctuation.definition.comment.end.custom": -1,
    "keyword.control.begin.custom": 1,
    "keyword.control.end.custom": -1
  },
  "punctuation": {
    "continuation": "punctuation.separator.continuation.line.custom"
  },
  "markers": {
    "start": "^\\s*#?region\\b",
    "end": "^\\s*#?end\\s?region\\b"
  },
  "symbols": {
    "keyword.control.custom": 2,
    "entity.name.function.custom": 11
  }
}

An example configuration file that targets Lua:

{
  "assignment": {
    "single": "meta.assignment.variable.single.lua",
    "multiple": "meta.assignment.variable.group.lua",
    "separator": "punctuation.separator.comma.lua"
  },
  "declarations": [
    "meta.declaration.lua entity.name",
    "meta.assignment.definition.lua entity.name"
  ],
  "dedentation": [
    "keyword.control.elseif.lua",
    "keyword.control.else.lua"
  ],
  "exclude": "{.luarocks,lua_modules}/**",
  "indentation": {
    "punctuation.definition.comment.begin.lua": 1,
    "punctuation.definition.comment.end.lua": -1,
    "keyword.control.begin.lua": 1,
    "keyword.control.end.lua": -1
  },
  "markers": {
    "start": "^\\s*#?region\\b",
    "end": "^\\s*#?end\\s?region\\b"
  },
  "symbols": {
    "keyword.control.lua": 2,
    "entity.name.function.lua": 11
  }
}

Usage

TextmateLanguageService

The package exports a default class named TextmateLanguageService.

  • Parameter: languageId - Language ID of grammar contribution in VS Code (string).
  • Parameter: context? - Extension context from activate entrypoint export (vscode.ExtensionContext).

The library defaults to core behaviour when figuring out which scope name to use - last matching grammar or language wins. If the context parameter is supplied, the extension will first search contributions from the extension itself.

Language extension

Extension code sample - ./src/extension.ts:

import TextmateLanguageService from 'vscode-textmate-languageservice';

export async function activate(context: vscode.ExtensionContext) {
	const selector: vscode.DocumentSelector = 'lua';
	const textmateService = new TextmateLanguageService(selector, context);

	const foldingRangeProvider = await textmateService.createFoldingRangeProvider();
	const documentSymbolProvider = await textmateService.createDocumentSymbolProvider();
	const workspaceSymbolProvider = await textmateService.createWorkspaceSymbolProvider();
	const definitionProvider = await textmateService.createDefinitionProvider();

	context.subscriptions.push(vscode.languages.registerDocumentSymbolProvider(selector, documentSymbolProvider));
	context.subscriptions.push(vscode.languages.registerFoldingRangeProvider(selector, foldingRangeProvider));
	context.subscriptions.push(vscode.languages.registerWorkspaceSymbolProvider(workspaceSymbolProvider));
	context.subscriptions.push(vscode.languages.registerDefinitionProvider(selector, peekDefinitionProvider));
};

Tokenization

Extension code sample - ./src/extension.ts:

import TextmateLanguageService from 'vscode-textmate-languageservice';

export async function activate(context: vscode.ExtensionContext) {
    const selector: vscode.DocumentSelector = 'custom';
    const textmateService = new TextmateLanguageService('custom', context);
    const textmateTokenService = await textmateService.initTokenService();
    const textDocument = vscode.window.activeTextEditor!.document;
    const tokens = textmateTokenService.fetch(textDocument);
};

NB: If you would like to:

You can use the custom "textmate-languageservice-contributes" property in package.json:

{
	"textmate-languageservice-contributes": {
		"languages": [{
			"id": "typescript",
			"aliases": ["TypeScript"],
			"extensions": [".ts", ".tsx", ".cts", ".mts"]
		}],
		"grammars": [{
			"language": "typescript",
			"scopeName": "source.ts",
			"path": "./syntaxes/TypeScript.tmLanguage.json"
		}]
	}
}

API methods

Usage (example is for getting the token at the current cursor position):

const { getScopeInformationAtPosition } = TextmateLanguageService.api;

const editor = vscode.window.activeTextEditor;
const document = editor.document;
const position = editor.selection.active;

const token = await getScopeInformationAtPosition(document, position);

getScopeInformationAtPosition

getScopeInformationAtPosition(document: vscode.TextDocument, position: vscode.Position): Promise<TextmateToken>

Get token scope information at a specific position (caret line and character number).

  • Parameter: document - Document to be tokenized (vscode.TextDocument).
  • Parameter: position - Zero-indexed caret position of token in document (vscode.Position).
  • Returns: Promise resolving to token data for scope selected by caret position ({Promise<TextmateToken>}).

getScopeRangeAtPosition

getScopeRangeAtPosition(document: vscode.TextDocument, position: vscode.Position): vscode.Range;

Get matching scope range of the Textmate token intersecting a caret position.

  • Parameter: document - Document to be tokenized (vscode.TextDocument).
  • Parameter: position - Zero-indexed caret position to intersect with (vscode.Position).
  • Returns: Promise resolving to character and line number of the range (Promise<vscode.Range>).

getTokenInformationAtPosition

getTokenInformationAtPosition(document: vscode.TextDocument, position: vscode.Position): Promise<vscode.TokenInformation>;

VS Code compatible performant API for token information at a caret position.

  • Parameter: document - Document to be tokenized (vscode.TextDocument).
  • Parameter: position - Zero-indexed caret position of token in document (vscode.Position).
  • Returns: Promise resolving to token data compatible with VS Code (Promise<vscode.TokenInformation>).

getLanguageConfiguration

getLanguageConfiguration(languageId: string): LanguageDefinition;

Get the language definition point of a language mode identifier.

  • Parameter: languageId - Language ID as shown in brackets in "Change Language Mode" panel (string).
  • Returns: Language contribution as configured in source VS Code extension (LanguageDefinition).

getGrammarContribution

getGrammarConfiguration(languageId: string): GrammarLanguageDefinition;

Get the grammar definition point of a language mode identifier.

  • Parameter: languageId - Language identifier, shown in brackets in "Change Language Mode" panel (string).
  • Returns: Grammar contribution as configured in source VS Code extension (GrammarLanguageDefinition).

getLanguageContribution

getLanguageConfiguration(languageId: string): LanguageDefinition;

Get the language configuration of a language mode identifier.

  • Parameter: languageId - Language ID as shown in brackets in "Change Language Mode" panel (string).
  • Returns: Language contribution as configured in source VS Code extension (LanguageDefinition).

getContributorExtension

getContributorExtension(languageId: string): vscode.Extension<unknown> | void;

Get the VS Code Extension API entry of the extension that contributed a language mode identifier.

  • Parameter: languageId - Language identifier, shown in brackets in "Change Language Mode" panel (string).
  • Returns: Extension API instance that contributed the language - (vscode.Extension).

Use Oniguruma WASM buffer

This is the vscode-oniguruma build of Oniguruma written in C, compiled to WASM format with memory hooks to V8.

This is not streaming 🙁 but vscode libs must bundle WebAssembly deps so as to support web ecosystem.

import TextmateLanguageService from 'vscode-textmate-languageservice';
const onigurumaPromise = TextmateLanguageService.utils.getOniguruma();