@anysphere/tiktoken-node
v0.2.1
Published
We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr
Downloads
4,141
Readme
@anysphere/tiktoken-node
We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thread.