Pandora Elasticsearch

Addon to provide flow and setup functions to Elasticsearch persistence for Pandora.

npm install --save @exocet/pandora-elasticsearch

Concepts

Native ID

There are two approaches to index documents into Elasticsearch: using custom IDs or Elasticsearch generated IDs. Elasticsearch have some performance issues with custom IDs and how they suffer from lookup into Lucene index. To work around this issue this addon have two working modes per entity:

• Native ID off - With native ID off the data will be indexed with custom IDs, normally UUIDv4's generetad at application side (this feature is supported by this addon with zero effort).
• Native ID on - The addon operates with Elasticsearch generated IDs and is going to create an id field in the source of the documents to put the application generated custom IDs (here we can use the UUIDv4 auto generation feature too).

Setup

Available features:

• Create indexes with custom configuration and dynamic aliases
• Create mappings from entities definitions
• Provides input, output and AQS query parsers
• Flow steps: insert (index), update, delete, exists and search

To add this addon into your project, put the addon spec into your endpoint YAML:

kind: Pandora/endpoint
metadata:
  name: myEndpoint
spec:
  addons:
    - name: elasticsearch
      package: "@exocet/pandora-elasticsearch"
      flow: true
      setup: true
  configuration:
    elasticsearch: # Any configuration here will be passed to client configuration (1)
      node: http://127.0.0.1:9200
      searchOptions: # Optional options applied in all searches
        maxPageSize: 50 # Maximum allowed search page size
      indices:
        - entityName: myNamespace.myEntity # To create index and mapping for an entity, pass the entity name, the settings bellow are the defaults
          numberOfShards: 1
          numberOfReplicas: 1
          autoExpandReplicas: false
          refreshInterval: 1s
          maxResultWindow: 10000
          maxInnerResultWindow: 100
          checkOnStartup: false
        - entityName: myNamespace.myEntityWithCustomIndex
          index: my_entity_with_custom_index # Custom entity index name instead of autogenerated
        - index: my_index # You can create custom indexes
          mapping: # With custom mappings
            myField:
              type: keyword
          aliases: # With custom aliases
            my_alias: true # Normal alias
            my_filtered_alias: # Filtered alias (2)
              term:
                active: true

1. Client configuration
2. Filtered alias

After the setup the following property of the context will be available:

• .application.dbs.elasticsearch - The client connected to Elasticsearch

Hooks

The hooks created by this addon are:

• elasticsearchQueryParsers (.service.hooks.useHook('elasticsearchQueryParsers')) - Synchronous hook that parses the search AQS query into Elasticsearch search body.

  const [queryParsers] = this.service.hooks.useHook('elasticsearchQueryParsers');
  const queryParser = queryParsers(entityName);
  const {
  	body, // The bodybuilder instance with filters from AQS query (1)
  	includes // The included fields requested by AQS query
  } = queryParser(
  	aqsQuery,
  	{ // Optional options
  		body: bodybuilder(), // Existent instance of bodybuilder to build the filters (1)
  		nativeId: false // Indicates if the targed entity uses native id approach
  	}
  );

• elasticsearchOutputParsers (.service.hooks.useHook('elasticsearchOutputParsers')) - Synchronous hook that parses the search output from Elasticsearch (each hit object) into entity representation.

  const [outputParsers] = this.service.hooks.useHook('elasticsearchOutputParsers');
  const outputParser = outputParsers(entityName);
  const results = response.body.hits.hits.map(outputParser);

• elasticsearchInputParsers (.service.hooks.useHook('elasticsearchInputParsers')) - Synchronous hook that parses the input from entity data into Elasticsearch document source representation.

  const [inputParsers] = this.service.hooks.useHook('elasticsearchInputParsers');
  const inputParser = inputParsers(entityName);
  const documentToIndex = inputParser(entity);

1. bodybuilder

Flow

The provided flow type and steps are listed bellow:

• insert - This flow step index data into Elasticsearch:

  kind: Pandora/flowStep
  metadata:
    name: entityPersistence
    labels:
      operation: insert
  spec:
    type: elasticsearch # Flow step type
    options:
      operation: insert # Defines that is an insert operation
      inboundFrom: request.data # The path to the data to be indexed (acquired from execution context)
      idFrom: null # If you want to use a custom ID instead of UUIDv4 you can pass the path from the execution context to get the ID value
      indexName: null # If you're using custom index name, pass the name here
      nativeId: false # Enables or disables the Native ID use
      entity: # The entity to define the input parser hook
        namespace: myNamespace
        name: myEntity

• exists - This flow step check if an entity was indexed by ID:

  kind: Pandora/flowStep
  metadata:
    name: entityPersistence
    labels:
      operation: exists
  spec:
    type: elasticsearch
    options:
      operation: exists
      idFrom: request.data.id # The path of the ID
      outboundTo: null # The execution path of the context to put the found ID, is an useful flow step to put before update and delete flows to get the native ID before operating the entity
      nativeId: false # This is important, this option will instruct the flow step to search by _id field or the _source.id field
      indexName: null
      entity:
        namespace: myNamespace
        name: myEntity

• update - This flow step updates indexed entities:

  kind: Pandora/flowStep
  metadata:
    name: entityPersistence
    labels:
      operation: update
  spec:
    type: elasticsearch
    options:
      operation: update
      inboundFrom: request.data
      idFrom: request.data.id # The path of the ID to update entity
      indexName: null
      entity:
        namespace: myNamespace
        name: myEntity

• delete - This flow step deletes indexed entities:

  kind: Pandora/flowStep
  metadata:
    name: entityPersistence
    labels:
      operation: delete
  spec:
    type: elasticsearch
    options:
      operation: delete
      idFrom: request.data.id
      indexName: null
      entity:
        namespace: myNamespace
        name: myEntity

• search - This flow step searches for indexed entities using AQS:

  kind: Pandora/flowStep
  metadata:
    name: entityPersistence
    labels:
      operation: search
  spec:
    type: elasticsearch
    options:
      operation: search
      queryFrom: request.query # The execution context path that have the AQS querystring
      outboundTo: response # The execution context path to put the search result
      bodyFrom: null # The execution context path that have an existing bodybuilder instance to append the parsed AQS filters, pass null to indicate the flow step to generate a new instance
      indexName: null
      entity: # Defines the AQS parser and the output parser
        namespace: myNamespace
        name: myEntity