batching-library

Library for batching in elastic.io. Batching allows users to combine several calls to API in one big call.

BatchClient

BatchClient represents client that works with batches. It contains logic for creating, updating statuses and storing batches.

Constructor

new BatchClient(store, config)

BatchClient constructor expects three parameters:

1. store - implementation of IStore interface. For example MongoStore. This is where batches will be stored.
2. config - instance of BatchConfig. It represent configuration and defines limitation for batches e.g: maxSize, maxItemsNumber.

Example

import { MongoStore, IStore } from '@elastic.io/batching-library';
import { BatchClient } from '@elastic.io/batching-library';
import { BatchConfig } from '@elastic.io/batching-library';
import { ConnectionOptions } from 'mongoose';

const maxSize = 1000;
const maxItemsNumber = 10;
const maxWaitTime = 1000;
const maxEmitRate = 1;
const maxRetry = 0;
const uri: string = 'url_to_mongo';
const collection = 'collection_name';
const connectionOpt: ConnectionOptions = {
  user: 'user',
  pass: 'password', 
};
const mgStore: IStore = new MongoStore(collection, uri, connectionOpt);
const config = new BatchConfig(maxSize, maxItemsNumber, maxWaitTime, maxEmitRate, maxRetry);
const client = new BatchClient(mgStore, config);

Methods

1. saveItem(item)

Save provided item to batch, returns batch with saved item (without another Batch items).

BatchItem properties

|Parameter|Type|Required|Description| |---------|----|--------|-----------| |id|string|false| If not specified - uuid v1 will be generated| |item|string|true| Body of the Batch Item|

Example

const batch: Batch = await client.saveItem({ id: 0, item: {}});

2. getReadyBatches()

Using provided emitter emit batches with status 'READY'

Example

const batches: Batch[] = await client.getReadyBatches();
  await Promise.all(batches.map(async (batch) => {
    try {
      
     // process batch implementation
      
      await client.updateBatchStatusById(batch.id, 'SUCCESS'); // batch was successfully processed
    } catch (e) {
      log.error('Error: %o', e);
      await client.updateBatchStatusById(batch.id, 'FAILED'); // batch processed with error
    }
  }));

Implementing your own batch storage

For reference implementation example take a look at MongoStore

1. Extend AbstractStore and override each abstract method.
2. Implementation must ensure that each operation is transactional

Environment variables

|Name|Mandatory|Description|Values| |----|---------|-----------|------| |LOG_OUTPUT_MODE| Yes| Log record view format | short long simple json bunyan| |LOG_LEVEL| Yes| Log Level | fatal error warn info debug trace | |MONGO_URL| No| URL of mongo db used during integration tests | url to mongo db| |MONGO_USER| No| Username of mongo db user, used during integration tests | username | |MONGO_PASSWORD| No| Password for connecting to mongo db, used during integration tests | password| |MONGO_DB| No| Database of mongo db used during integration tests | database name | |MONGO_DROP_DB| No| If 1 drop database each test integration test if 0 dont | 1 0|

Environment template file

Limitations

MongoStore:

1. Supported Mongo database versions: 4.0 and higher.

2. Retries do not implemented.

3. Emit rate not implemented.

4. Library is not guarantee sequence processing of batch item

MaesterStore:

1. Emit rate not implemented.