@idempotender/core
v0.1.4
Published
A Javascript library for idempotency control based on DynamoDB
Downloads
1
Readme
idempotender core
This is the javascript library with basic functions for implementing a Idempotent function.
It is able to acquire a lock for an execution to prevent concurrency situations, store and retrieves states from DynamoDB between executions based on a key.
Usage
npm install --save @idempotender/core
Create DynamoDB table with structure:
Resources:
IdempotencyExecutions:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: Id
AttributeType: S
KeySchema:
- AttributeName: Id
KeyType: HASH
TimeToLiveSpecification:
AttributeName: executionTTL
Enabled: true
Example: Simplest form
- Create function
import { withIdempotency } from '@idempotender/core';
function myIdempotentFunction(param1: string, param2: string): string {
const out = await withIdempotency(():string => {
// business logic
return `First run at ${new Date()}`;
},
// idempotency key
`${param1}:${param2}`);
return out;
}
Example: Using more advanced options of the API
It will deactivate the hashing mechanism, so in the database you can see the actual contents of param1:param2 and there is no chance of collision.
Create function
import idempotender from '@idempotender/core';
const idem = idempotender({
executionTTL: 24 * 3600,
dynamoDBTableName: 'IdempotencyExecutions',
keyHash: false,
});
function myIdempotentFunction(param1: string, param2: string): string {
// get current idempotency status
// acquire lock to avoid concurrent calls to this function
const execution = await idem.getExecution(`${param1}:${param2}`);
// already processed before, so return previous output data response
if (execution.statusCompleted()) {
return execution.output();
}
// another process is processing this in parallel
if (execution.statusLocked()) {
throw new Error(`Concurrent processing for idempotency key ${execution.key}`);
}
// status is "open", so
// DO BUSINESS HERE
try {
console.log(`I only hello world for (${param1},${param2}) once!`);
const output = `This is output '${param1}${param2}' at ${new Date()}`;
} catch (err) {
console.log('Error during function execution');
idem.cancel(execution.key);
}
// save execution output so further calls in the next 24h
// using the same "param1,param2" as key will return this response
// without actually running the business function
idem.complete(execution.key, output);
return output;
}
Reference
- These are the default values of the configuration
const idem = idempotender({
dynamoDBTableName: 'IdempotencyExecutions',
lockEnable: true,
lockTTL: 10,
lockRetryTimeout: 15,
executionTTL: 24 * 3600,
keyHash: true
}
Config attributes:
dynamoDBTableName
DynamoDB table to use for controlling idempotency
Defaults to 'IdempotencyExecutions'
lockEnable
Avoids parallel concurrency situations in which, while the first execution is running, another request arrives (before the first is finished)
In this situation, the second request will be responded with error 409 (conflict) so the client can resubmit it again later (and then receive a valid response, cached from the first run - because of the idempotency)
Activating this may increase your DynamoDB costs by ~3x, but is recommended because it's safer
Defaults to 'true'
lockTTL
Time in seconds that the concurrency lock will be active. During this timespan, if another request arrives, before the first request is finished, it will receive a status 419 (see lockEnable)
Defaults to '10'
lockRetryTimeout
Time in seconds waiting for an existing lock to be released in case of concurrency. If the lock is released in this period, we will try to get the last saved output from the other process (if it was saved) and can return the previous output gracefully. If after lock is released and output was not saved, we will try to acquire the lock again.
Defaults to '15'
executionTTL
Time in seconds after execution is finished in which if another request with the same key arrives, will be responded with the first execution response.
The actual execution will be skipped, but the client will receive the same response as in the first call.
Defaults to '24 * 3600'
keyHash
Whatever hash the key before storing in the database or not
Useful in cases where the key is too large or you don't want to expose the input parameters in plain text in DynamoDB
This is applied after keyMapper is executed
SHA-256 is used, which is very secure, but keep in mind that there is a very low possibility of collisions if this is enabled.
Defaults to 'true'
Functions
completeExecution(key:string, output:string)
- Save the output as the execution result for a specific key, so that next time anyone tries to "getExecution(key)" with this key, it will be returned
getExecution(input:string):object
Queries the execution table for a certain input. If it doesn't exist and lock is enabled, creates the record for lock control
The actual key used in database operations is the result of the 'input' mapped to a key (see config 'keyMapper' and 'keyJmespath') and, if enabled, hashed with SHA-256.
Returns an object with struct:
statusCompleted():boolean
- Returns true if a previous execution with corresponding output data was saved and is available via attribute 'output'
statusLocked():boolean
- Returns true if another function started processing this but didn't reached "saveExecution()" still, indicating these two executions might be running in parallel, but only one should be able to actually run
output: previously saved output, indicating this function was already called before
deleteExecution(key:string)
- Deletes an execution. Normally used when something goes wrong during the function execution and you want to clear the execution so another call can try to execute the function again later on