npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

es-odm

v6.2.0

Published

ODM

Downloads

114

Readme

ElasticSearch 8 Node.js ODM library

Basic information

  • This library maps ES documents and functions to JS classes

Alias

  • ODM models are represented via ES aliases

  • Calling functions like createIndex() will create an underlying index and also sets an alias into it

  • Most of the communication is accomplished via aliases, real indices are used only when necessary

    • This is to simplify some processes like reindexing/cloning/migrating from source (old) index into the new one
      • Thanks to this, we can create new index, migrate data from the original one, than switch the aliases and delete the original one.
      • And the index are still in human-readable format
  • Alias consist from two parts:

    • tenant (AKA prefix) - defaults to *
    • name - main index name
  • Final alias looks like this:

    • tenant_name

Work with aliases

  • Alias name is specified when model is created, like this - createClass('myIndex')
  • Tenant can be specified using createClass('myIndex', void 0, 'myTenant')
    • or later by using .in('myTenant')
  • Function in('tenant') (and immediateRefresh(bool) as well) creates new ODM model with updated values, original ODM remains unchanged
  • example:
    • createClass('myIndex').in('default') -> default_myIndex
    • createClass('myIndex') -> *_myIndex
    • createClass('myIndex', void 0, 'tenant') -> tenant_myIndex
    • createClass('myIndex', void 0, 'tenant').in('changedTenant') -> changedTenant_myIndex
  • When creating a new instance (or performing some operations), the alias cannot contain any wildcard
    • it means you must fully specify the tenant

Underlying index

  • Underlying index is in a very similar format, just also contains unique identifier:

    • tenant_name-id
  • This library is made to use aliases, but technically speaking it should be also working with ES indices directly (or even with mixture of indices and aliases)

    • It is necessary to avoid any conflicts between indices / aliases
    • Newly created indices will always be created as aliases with underlying indices

Usage

  • const { createClass, BulkArray, BaseModel, JointModel, esClient, setClient, esErrors, setLoggerConfig, setLoggerUidFunction } = require('odm');
  • createClass function creates new class
    • Each class contains several properties
      • schema, alias, _tenant, _name and _immediateRefresh
    • Class static functions profit from those properties
      • e.g. static search() uses them to get alias
      • You may write your own static function, which will use those properties
        • Or rewrite existing functions
          • Which may influence another functions
        • Or redefine class properties
    • Instances have access to those properties / functions
      • myInstance.constructor.alias
      • And they use them during ES operations
        • eq. to validate data, ...
    • Whenever we need new / independent class, we must either create a new one or clone an existing one
  • Additionally, library provides BulkArray
    • Class inherited from Array
    • Contains methods which uses ES Bulk API
      • save, delete and reload
      • Plus some additional methods to dynamically mark and remove ODM instance - see in code
  • Exported BaseModel should be used mainly for instanceof checks
    • Do not create instances from it / do not change its parameters (unless you really know what you are doing)
  • JointModel is a special class used to "record" searches from multiple distinct ODM classes and then perform single search with different queries above multiple indices
  • setClient replaces ES client singleton
    • Should be called once at application startup
    • New client is then used - even in already created classes
  • esErrors - Errors from underlying ES library
  • setLoggerConfig - Replaces default logger configuration
  • setLoggerUidFunction - Pass custom function to be used to generate ID, which is passed to generated logs

Class usage

Create new class
  • To create new class, call createClass(name, ?schema, ?tenant):
    • name - essential, it must be specified
    • schema - may be undefined or Joi object, it is there, so we can do additional validations
    • tenant - Serves as part of the index (prefix)
      • This is required and cannot contain underscores (_)
      • By default, it is *
        • You can rewrite it later by using .in('newTenant')
          • It will create inherited class
          • const SpecificTenantClass = MyClass.in('newTenant');
        • You must specify it before creating new instance
        • When searching for records, found records will have correctly set tenant
Modify / clone existing class
  • Class can be modified / cloned using:
    • in(tenant)
      • Returns class clone with tenant changed to given value
    • immediateRefresh(bool)
      • Returns class clone with immediate refresh set to given value
        • By default, new classes (and its instances) does perform refresh: true with all write operations -> You can use this to disable it
    • clone(?changes)
      • Clones class
      • changes is optional object to set cloned class properties
      • This method is internally used by in() and immediateRefresh()
      • You should not need to use it
  • Cloning class means:
    • New inherited class is created
    • Changes made afterwards to cloned class are NOT transferred to original one

Instances

  • Instances are made from prepared class
    • Manually:
      • You prepare class and then you can call new MyClass(?data, ?_id, ?_version, ?_highlight, ?_primary_term, ?_seq_no, ?_score, ?_sort)
        • data is optional object whose properties are placed to instance
        • _id is optional ES _id
        • _version is optional ES _version
        • _highlight is optional ES highlight
        • _primary_term is optional ES _primary_term
        • _seq_no is optional ES _seq_no
        • _score is optional ES _score
        • _sort is optional ES sort
    • From static functions:
      • When you call functions like findAll(), search(), ...
        • Instance is made from class and correct data is loaded from ES
  • Instance contains only ES data and methods to save / reload / validate / ...
    • All ES properties, search functions, ... are saved in class
  • NOTE: _id, _version, _highlight, _primary_term, _seq_no, _score and _sort are not enumerable

BulkArray

  • For ES Bulk API, you can use BulkArray
  • Class inherited from Array
  • All static search functions in BaseModel class returns BulkArray of instances instead of Array
  • Provides bulk functions:
    • async save(?useVersion)
      • Saves all items to ES
        • Not existing ids are generated and pushed to instances
      • Returns ES response
      • useVersion - Checks if version match
        • uses sequence numbers internally, if not presented, it will fetch them and checks version automatically
    • async delete(?useVersion)
      • Deletes all items from ES
      • Returns ES response
      • useVersion - Checks if version match - uses sequence numbers internally, if not presented, it will fetch them and checks version automatically
    • async reload()
      • Reloads all instances in bulk array

Examples:

class Counter extends createClass('counter', Joi.object()) {
  async customFunction() {
      return 666;
  }
  
  static customStatic() {
      return new this({ a: 4 }, `myId`);
  }
  
  static async search(...args) {
      // Rewritting static function -> affects search, find and findAll
      const results = await super.search(...args);
      if (results.length <= 0) {
          throw Error(`Nothing`);
      } else {
          return results;
      }
  }
}

...

const MyCounter = Counter.in('default'); //Index is 'default_counter'

const allCounters = await MyCounter.findAll();    //BulkArray with all counters in index
await allCounters.delete();   //Everything deleted from ES

const instanceCounter = new MyCounter({ counter: 15 }, 'myId');   //Prepared new instance
await instanceCounter.save(); //New instance saved

Functions / Methods

Internal getters
  • static get alias
    • Returns class full alias, usable for ES queries
    • It consists of tenant and name
Class level API
  • static async search(body, ?from, ?size, ?additional)

    • Performs ES search
    • Returns BulkArray of instances
    • Used by another static functions
      • Redefining this function will affect their behavior
    • User must specify body
      • alias is already defined in the class
    • from and size are optional
      • Returns all results if from / size are not specified, no matter how many there are
    • additional is an optional object with additional data
  • static async *bulkIterator(body, ?source, ?bulkSize, ?additional)

    • Async generator for iterating over bulks of documents
    • body, source and additional arguments are the same as in the search function
    • bulkSize is optional custom size of a bulk
  • static async *itemIterator(body, ?source, ?bulkSize, ?additional)

    • Async generator for iterating over documents
    • Arguments are the same as in the bulkIterator generator
  • static async findAll()

    • Finds all entries in ES, matching class alias
    • Uses this.search()
      • Returns BulkArray
  • static async find(ids)

    • Performs ES 'search' query
    • Always returns BulkArray of instances
    • Uses this.search()
  • static async get(ids)

    • Performs ES 'get'
    • If 'ids' is strings, returns single instance
    • Else if 'ids' is an array of strings, returns BulkArray of instances
  • static async head(ids)

    • Performs ES 'mget' without '_source' field
    • Throws when any of the IDs is not found
    • If 'ids' is strings, returns single object
    • Else if 'ids' is an array of strings, returns array of objects
  • static async delete(ids, ?version)

    • Performs ES 'delete'
    • Uses bulk API
    • Class alias must be fully specified
    • Returns ES response
    • If version is specified, only one id is allowed
  • static async exists(ids)

    • Performs ES 'exists'
    • If 'ids' is strings, returns single boolean
    • Else if 'ids' is array of strings, returns array of booleans
      • true if exists, else otherwise
  • static async update(ids, body)

    • Performs ES 'update'
    • Uses bulk API
    • Returns ES response
  • static async count(?body)

    • Performs ES 'count'
    • Returns number
  • static async updateByQuery(body)

    • Performs ES 'update_by_query'
    • Returns ES response
  • static async deleteByQuery(body)

    • Performs ES 'delete_by_query'
    • Returns ES response
Indices API
  • static async createIndex(?body, ?setAlias = true)

    • Creates index given by current class
    • body is optional settings
    • setAlias is used when we do not want automatically set an alias to newly create index
  • static async getIndex()

    • Returns ES index of ODM alias
    • Returns string
  • static async aliasIndex(index)

    • Puts alias of current ODM onto selected index
  • static async deleteAlias()

    • Deletes ODM alias from ES, underlying index remains unchanged
  • static async aliasExists()

    • Checks ODM alias existence
    • Returns boolean
  • static async indexExists()

    • Checks ODM index existence
    • Returns boolean
  • static async deleteIndex()

    • Deletes alias and index given by current class
  • static async getMapping()

    • Gets mapping of index given by current class
  • static async putMapping(mapping)

    • Puts mapping to index given by current class
  • static async getSettings(?includeDefaults = false)

    • Gets settings of index given by current class
  • static async putSettings(settings)

    • Puts settings to index given by current class
  • static async reindex(destinationModel, ?script)

    • Reindex from current model class to selected one
    • Destination may be chosen by ODM or by alias/index string
    • script specifies optional painless script to be used
  • static async cloneIndex(?settings)

    • Creates clone of ODM index
    • Current index has to be manually blocked for write (be made read-only)
    • settings is optional settings to be used for newly created index
    • Returns newly created index
  • static async refresh()

    • Refreshed current index / indices
  • static async openPIT()

    • Opens Point in Time in given index
  • static async closePIT(id)

    • Closes given Point in Time
Instance level API
  • async save(?useVersion)
    • saves or re-saves instance
    • it uses specified _id or generates new one if not specified
      • it uses ES 'index' function
    • useVersion - checks if version matches, _id and _version must be specified
      • sequence number will be fetched automatically in not presented
  • async reload()
    • Reloads instance data from ES
    • Uses get and new this.constructor() internally
  • async delete(?useVersion)
    • Deletes an instance from ES
    • _id must be specified and entry must exist in ES
    • useVersion - checks if version matches, _version must be specified
      • sequence number will be fetched automatically in not presented
  • clone(?_preserveAttributes)
    • Returns clone of current instance
      • Deep copy
    • Cloned instance is created from the same class
    • preserveAttributes (defaults true) can be used to preserve attributes (_id, _version, ...)
  • async validate()
    • Validates instance using Joi
    • Throws in case of error / incorrect data
Class copy
  • static clone(?changes)
    • Creates class copy
    • In fact, it creates new class which inherits from the current one
    • changes is an optional object with properties to be set
  • static in(newTenant)
    • Clones class using clone()
    • Sets given tenant
  • static immediateRefresh(newRefresh)
    • Clones class using clone()
    • Sets given refresh mode

Other

  • static _parseIndex(index)

    • Parses given index (or even alias) and returns its parts
  • static __checkIfFullySpecified(functionName)

    • Checks if current class is fully specified, meaning it doesn't contain any wildcard in alias
    • Throws otherwise
  • static __getConstructor(searchResult, constructorCache)

    • Returns the correct class (constructor) for given search result
  • static _alterSearch(body)

    • Alters the search function query (and also query of other functions that use the query)
    • By default, just passes the body without any change, but can be rewritten
  • static _unpackData(source)

    • Alters the fetched ES documents before the instance is created from them
    • By default, just passes the source without any change, but can be rewritten
  • async _packData(cache)

    • Alters the instance data before it is saved to ES
    • By default, just passes the instance data without any change, but can be rewritten
  • static async _afterSearch(instances, ?cache)

    • Special function called for each record founded by search / find / findAll / get
    • By default, it is empty, but you can overwrite it in your code
  • static async _getBulkSize()

    • Returns optimal size of bulk for given model
    • Used in search function for pagination