npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

clientside-search

v1.8.1

Published

A highly efficient, isomorphic, full-featured, multilingual text search engine library, providing full-text search, fuzzy matching, phonetic scoring, document indexing and more, with micro JSON state hydration/dehydration in-browser and server-side.

Downloads

1,792

Readme

"Why don't we have a decent, Lucene-like client-side (in-browser) search engine by now?"

This library provides Lucene-like full-text search features for the browser and Node.js.

This search engine uses several advanced algorithms to provide robust and efficient searching over a large collection of documents. The algorithms used include TF-IDF for weighing and ranking, BK-Tree for fuzzy matching, BM25 for relevance scoring, and Damerau-Levenshtein distance for measuring the edit distance between search terms. The search engine supports multiple languages and uses stemming and stopword removal to enhance its efficiency. It also supports the storage and retrieval of metadata associated with the documents. You can generate an index from a text corpus and metadata both on client- and server-side. You can hydrate and re-hydrate (reuse a pre-generated) the index as well on both client- and server-side.

  1. I want to use a Lucene-like index that uses TF-IDF vectorization, BM25 and BKTree ranking as well as snowball stemming by and stopwords on client side.

  2. I want to generate the search index either on client side or server-side (and re-hydrate/re-use it on client or server-side). State information should be small and compressed.

  3. The full-text search shall be fast and efficient, not leading to alot of false-positives or false-negatives.

  4. The search engine should be able to retreive and search in metadata that may be associated with each document.

  5. The search engine should be able to remove/update it's index' documents.

  6. State shall be hydratable.

  • Full-Text Search: Provides the ability to perform a comprehensive text-based search over a large collection of documents.
  • Multilingual Support: Supports multiple languages for indexing and searching documents and automatic language detection for input text.
  • Text Processing: Includes text transformation operations like converting to lower case, splitting by words, snowball stemming (Dr. Martin Porter), and stopword removal.
  • Document Indexing: Allows adding of documents to the index along with metadata to make them searchable. Document Removal: Provides functionality to remove a specific document from the index based on its ID.
  • Search Query Processing: Processes search queries in the same way as document text to ensure a consistent matching algorithm.
  • Relevance Scoring with BM25 Algorithm: Uses the BM25 algorithm for relevance scoring of documents against search queries.
  • Fuzzy Matching with BK-Tree: Uses a BK-Tree structure to perform fuzzy matching, i.e., to find words in the index that are similar to the search terms.
  • Term Frequency-Inverse Document Frequency (TF-IDF) Weighting: Uses TF-IDF to weight and rank the indexed words based on their importance in the document and rarity in the overall document set.
  • State Hydration and Dehydration: Provides functionality to save (dehydrate) the state of the search engine to a compressed format, or to restore (hydrate) it from a previously saved state either locally or remotely.
  • Damerau-Levenshtein Distance Calculation: Includes a function to calculate the Damerau-Levenshtein distance, i.e., the minimum number of operations (insertions, deletions, substitutions, transposition) required to change one word into another.
  • Phonetic Scoring: Uses language-specific phonetic algorithms such as Double Metaphone and Koelner Phonetik as a tie breaker when Damerau-Levenshtein Distance is equal for two matches.
  • Document ID Generation: Generates a unique ID for each document based on its text.
  • Automatic Stop Word selection: Selects the best default stop words per language supported.
  • ✅ Currently supports only: en, de, fr, es, ja
  • ✅ Supports UTF8
  • ✅ Available as a simple API
  • ✅ Just 8 KiB nano sized (ESM, gizpped, base library)
  • ✅ Zero dependencies!
  • ✅ Tree-shakable and side-effect free
  • ✅ First class TypeScript support
  • ✅ Well tested using Jest Unit Tests
  • yarn: yarn add clientside-search
  • npm: npm install clientside-search
import { SearchEngine } from 'clientside-search'
import en from 'clientside-search/en'

// create a new instance of a search engine
const searchEngine = new SearchEngine(en)

// add some text
const docId1 = searchEngine.addDocument('The quick brown fox jumps over the lazy dog')

// you can also add UTF8 text, and metadata
const docId2 = searchEngine.addDocument('The quick brown fox jumps over the fence ✅', {
  // metadata with index_ prefix will be indexed for search
  index_title: 'Fence',
  date: new Date(),
  author: 'John Doe',
})

/**
 * {
 *   id:
 *   score: 1.34,
 *   metadata: { title: 'Fence', date: '2023-07-12 ...', author: 'John Doe' }
 * }
 */
const searchResult = searchEngine.search('Fence')

// if you want to persist the index state,
// hydratedState is a JSON string that you can persist
const hydratedState = searchEngine.hydrateState()

// PLEASE NOTE: The hydrated state does NOT contain the original input text
// It contains an optimized representation of the search index
// However, metadata is kept 1:1

// you can re-hydrate from that state anywhere,
// on the server or the client:
const hydratedEngine = SearchEngine.fromHydratedState(hydratedState, en)

// equals: searchResult
const searchResultFromHydated = hydratedEngine.search('Fence')
const { SearchEngine } = require('clientside-search')
const { en } = require('clientside-search/en')

// same API like ESM variant
  • Advanced Asian language support:
    • Support for Chinese using Jieba
      • No BKTree, but N-gram comparison
      • Character-based TF-IDF
      • Disable stemming
      • e.g. Jaccard similarity
    • Korean
      • No BKTree, but N-gram comparison
      • Jamo Levenshtein Distance
      • TF-IDF