Usage

npm i gpt-token

import { textTokens, messageTokens } from 'gpt-token'

// => 9
console.log(textTokens('A quick brown fox jumps over the lazy dog'))

// => 16
console.log(messageTokens([
  {
    role: 'user',
    content: 'A quick brown fox jumps over the lazy dog'
  }
]))

Thanks ❤️

• gpt-tokens - inspired by this project
• openai-cookbook - official tokenizer rules
• openai-tokenizer - official tokenizer playground

License

MIT License © Krcisleo