npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

humane-discovery

v2.0.5

Published

Humane Discovery

Downloads

30

Readme

Humane Discovery 1.1.9

Humane Discovery product stack aims to solve entire discovery problem space for content (such as exams, jobs), news & media, e-commerce, digital goods products etc. More specifically product approaches the discovery problem in following broad parts --

  • Search based discovery, such as autocomplete, suggested queries, search results, instance searches.
  • Personalised discovery such as recommended products (or posts).
  • Curated discovery such as curated collections or products.
  • Domain specific discovery algorithms such as news one may have missed since last visit.

We call the lowest unit that can be searched, personalised or curated a 'Product'. These can be any of the following --

  • Banners
  • Tabs
  • Categories
  • E-commerce Products
  • Digital Products such as e-books, PDFs
  • Posts such as News, Articles
  • Blog or Articles
  • Discounts
  • Offers
  • Delivery Slots
  • Collections

Search based discovery

  • APIs for
    • Autocomplete,
    • Suggested Queries,
    • Search Results,
    • Instance Search (Search As You Type).
  • Excellent support for fuzzy queries, and vernacular queries.
  • Easy synonyms management (add, update, & remove) without worrying about re-indexing as product takes care of re-indexing or query-expansion itself.
  • Match relevancy algorithm that considers
    • Some fields are more important than others.
    • Matches in single field are better than matches in two separate fields.
    • Phrase matches are better than single field matches.
    • Single field matches in lower weight field are better than matches in two different higher weight field.
  • Product's weight auto-computation from disparate type of signals - views, downloads, add to cart, cart abandonment, purchases, time spent, or any custom events.
  • Multiple ways to compute weight --
    • Weighted signals based weight.
    • Holt Winter's based trend forecast weight.

Personalised discovery [Not Yet Open Sourced]

  • Personalises entire experience based on user profile, past behavior, and geo-location, such as --
    • Banners
    • Tabs
    • Categories
    • E-commerce Products
    • Digital Products such as e-books, PDFs
    • Posts such as News, Articles
    • Blogs or Articles
    • Discounts
    • Offers
    • Delivery Slots
    • Collections
  • Auto learns user preferences based on implicit behaviors.
  • Auto A/B tests new products to learn user preferences.
  • Careful mix of recommendations, curated, A/B test products.
  • Collaborative filtering based discovery algorithms, such as people who bought (or any other signal) X also bought (or any other signal)

Curated discovery [Not Yet Open Sourced]

  • Hand or community curated products or their collections.

Domain specific discovery algorithms [Not Yet Open Sourced]

  • Many domain specific discovery algorithms, such as --
    • News one may have missed since last visit.
    • Popular Products
    • Trending Products
  • Custom algorithms can be composed in functional style.

Data Pipeline

  • A very powerful data pipeline that can import data from variety of sources --
    • CSV Files
    • TSV Files
    • JSON Files
    • Array of JSON
  • Transform or multiplex data through JS functions.
  • Extend data with lookups & mapper
  • Output data to Indexer, as of now (more to come).

How to Install

Follow prerequisites and then install it as global npm module: npm install -g humane-discovery

Prerequisites

  • node - v4.0.0+
  • npm - v2.0.0+
  • You are familiar with npm and know whether or not you need to use sudo when installing packages globally.
  • git - v2.7.0+
  • elasticsearch -v2.1+
  • analysis-phonetic plugin for elasticsearch - <es>/bin/plugin install analysis-phonetic

How to Run

Humane-discovery has two commands - one for cli, another for server

Running Server

Server runs the searcher and indexer API services and cockpit UI.

humane-server --discoveryPlugin <plugin module> --port [PORT] --transliterator [transliterator module]

Below are details on various options -

discoveryPlugin  - Path to single or list of comma separated discovery plugins.
                   Can be name of globally installed module or full path to module directory.
                   Defaults to:
                       1) Current directory - must be a valid node module
                       2) HUMANE_PLUGIN_DISCOVERY environment variable
                       
transliterator   - Path to transliterator plugin (optional).
                   Can be name of globally installed plugin module or full path to plugin directory.
                   Defaults to: HUMANE_PLUGIN_TRANSLITERATOR environment variable
                   
port             - Server Port. Defaults to: 3000                   

Running Cli

CLI provides various utility commands for managing indexes - such as adding (or removing) index, add (or update, delete, upsert) data, import data into index through data pipeline.

humane-cli --discoveryPlugin <plugin module>

Below are details on various options.

discoveryPlugin  - Path to single discovery plugin.
                   Can be name of globally installed module or full path to module directory.
                   Defaults to:
                       1) Current directory - must be a valid node module
                       2) HUMANE_PLUGIN_DISCOVERY environment variable

CLI is a meta-cli, that is CLI generates another CLI based on provided discovery plugin. This way commands are specific to the configured indexes, types, data pipeline. Below is such a sample CLI.

Sample CLI

How to Create Discovery Plugin

  1. Create a new npm module for the plugin.
  2. Add dependency on humane-indexer: npm install --save humane-indexer
  3. Create IndicesConfig.js, SearchConfig.js, and CockpitConfig.js - refer below for details
  1. You can optionally define DataPipelineConfig.js - refer Indexer Configuration for details.
  2. Create a index.js file.
  3. Export following from index.js
  • name of the instance (say exams) as instanceName
  • IndicesConfig.js as indicesConfig
  • SearchConfig.js as searchConfig
  • CockpitConfig.js as cockpitConfig
  • DataPipelineConfig.js as dataPipelineConfig

Sample index.js -

exports.instanceName = 'exams';
exports.dataPipelineConfig = require('./DataPipelineConfig.js');
exports.indicesConfig = require('./IndicesConfig.js');
exports.searchConfig = require('./SearchConfig.js');
exports.cockpitConfig = require('./CockpitConfig.js');
  1. In package.json specify this as 'main':
"main": "./lib/index.js"
  1. If you use ES6, main should refer to babel compiled code for es5. Use following babel presets -
  "babel": {
      "plugins": [
          "transform-runtime"
      ],
      "presets": [
          "es2015"
      ]
  }

API Documentation