@moveyourdigital/nlp
v0.1.4
Published
NLP (natural language processing) for server and the browser in TypeScript. All lightweight and super-fast.
Downloads
12
Maintainers
Readme
nlp
Natural Language Processing (NLP) written in TypeScript for both server and browser environments. Lightweight and ultra-fast.
Introduction
The goal of this package is to provide a lightweight and super-fast solution for natural language processing tasks.
It aims to be the go-to choice for developers seeking a powerful yet easy-to-use NLP solution in TypeScript. Whether working on the server-side or in the browser, this is optimized for performance and simplicity.
This package was inspired in the good but old apparatus
package.
Key Features:
Naive Bayes Classifier (BayesClassifier
class)
- A simple and effective classifier based on Bayes' theorem, ideal for text classification tasks.
- Supports serialization to JSON for easy model saving and sharing.
- Includes a compiled version where the corpus is stripped from the serialization output, reducing the model size for distribution when used solely for text classification (i.e., no further training).
RegexpTokenizer
- A versatile tokenizer with current support for English and Portuguese. This tool efficiently splits text into meaningful tokens using regular expressions.
Normalizer
- This feature allows for text normalization, accepting a stemmer and a list of stopwords. It integrates seamlessly with the
natural
npm package for stemming and thestopword
npm package for managing stopword lists.
Installation
npm i @moveyourdigital/nlp
Basic Usage
This is a minimal usage of the Naive Bayes classifier using sentences.
import { BayesClassifier, Tokenizer, WordsEnTokenizer } from '@moveyourdigital/nlp'
const tokenizer = new WordsEnTokenizer()
// or WordsPtTokenizer
const classifier = new BayesClassifier<string, string>()
.set('smoothing', 0.1)
.addDocument({
observation: tokenizer.tokenize(
'you are a beautiful person'
),
label: 'good',
})
.addDocument({
observation: tokenizer.tokenize(
'he is an evil person'
),
label: 'bad',
})
.train()
const classification = classifier.classify(
tokenizer.tokenize('i am a beautiful person')
)
Classification should output
[
{ label: 'good', score: 4.6305 }, // ordered by highiest score
{ label: 'bad', score: 1.2705000000000002 }
]
Serialize / Deserialize
The classifier supports flexible serialization and deserialization, making it easy to save, share, and deploy trained models.
Standard Serialization
One can serialize the classifier to a JSON string, enabling to store or transfer the model. This is especially useful for deploying models to different environments or saving the state of a model for future use.
const jsonString = classifier.toJSON();
// Standard JSON string representation of the model
Compact Serialization
For scenarios where the model is only needed for text classification (i.e., no further training is required), you can use the compact serialization option. This removes the corpus from the serialization output, significantly reducing the model's size for efficient distribution.
const jsonString = classifier.toJSON({ compact: true });
// JSON string representation of the model for distribution
Deserialization
To restore the model for further training or usage, deserialize the JSON string back into the classifier. If you intend to continue training the model, do not use the compact flag during serialization.
classifier.restore(jsonString); // Accepts both stringified or already parsed JSON
This flexible serialization and deserialization process ensures that NLP models can be managed efficiently, whether a lightweight distribution is needed or the full model for ongoing development.
Roadmap for further development
- Porter stemming
- Lemmatization
- TF-IDF (using BM25 extension)
- Support for new languages (help appreciated)
Development
To watch for changes in src
directory.
npm start
To bundle for production
npm run build
Unit tests are located in
npm t