@mlc-ai/web-tokenizers
v0.1.5
Published
| [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) |
Downloads
1,091
Readme
WebTokenizers
| NPM Package | WebLLM |
WebTokenizers is a javascript binding library that can be universally deployed. It wraps and binds the HuggingFace tokenizers library and sentencepiece and provides a minimum common interface in havascript.
Build from Srouce
source /path/to/emsdk_env.sh
npm install
npm run build
To try out the test webpage
cd tests
npm start