@lordluceus/mistral-tokenizer
v1.2.2
Published
TS tokenizer for Mistral-based LLMs
Downloads
65
Readme
🌬️ mistral-tokenizer-ts 🌬️
This repository is a TypeScript fork of mistral-tokenizer-js
, initially developed by imoneoi.
It is designed to maintain the core functionality of the tokenizer while leveraging the advantages of TypeScript for better type safety and developer experience.
Please refer to the original repository for more context and the foundational work behind the tokenizer.
Install
npm install mistral-tokenizer-ts
Usage
import { MistralTokenizer } from 'mistral-tokenizer-ts'
const tokenizer = new MistralTokenizer()
// Encode.
const tokens = tokenizer.encode('Hello world!')
// Decode.
const decoded = tokenizer.decode([1, 22557, 1526, 28808])
Tests
npm run test