@flexpilot-ai/tokenizers.js
v1.0.2
Published
JavaScript implementation of Hugging Face's tokenizers based on transformers.js
Downloads
56
Maintainers
Readme
This repository provides a lightweight, pure JavaScript implementation of Hugging Face's tokenizers. It is based on the tokenizers available in the transformers.js library. By removing dependencies such as ONNX and others, this library focuses solely on efficient text tokenization, offering a streamlined solution without the overhead of additional dependencies.
This project is ideal for those who require a simple and efficient way to tokenize text data using Hugging Face's tokenizers in JavaScript environments, without the need for heavy or unnecessary components.
Table of Contents
Installation
You can install the package via npm:
npm install @flexpilot-ai/tokenizers.js
Usage
Here is a basic example of how to use the tokenizer:
import { AutoTokenizer } from "@flexpilot-ai/tokenizers.js";
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/bert-base-uncased');
const { input_ids } = await tokenizer('I love tokenizers.js!');
Issue Tracking
If you encounter any issues, please report them here.
License
This project is licensed under the Apache-2.0 License. See the LICENSE
file for details.