npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@idempotender/core

v0.1.4

Published

A Javascript library for idempotency control based on DynamoDB

Downloads

1

Readme

idempotender core

This is the javascript library with basic functions for implementing a Idempotent function.

It is able to acquire a lock for an execution to prevent concurrency situations, store and retrieves states from DynamoDB between executions based on a key.

Usage

  • npm install --save @idempotender/core

  • Create DynamoDB table with structure:

Resources:
  IdempotencyExecutions:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: Id
          AttributeType: S
      KeySchema:
        - AttributeName: Id
          KeyType: HASH
      TimeToLiveSpecification:
        AttributeName: executionTTL
        Enabled: true

Example: Simplest form

  • Create function
import { withIdempotency } from '@idempotender/core';

function myIdempotentFunction(param1: string, param2: string): string {

  const out = await withIdempotency(():string => {
    // business logic
    return `First run at ${new Date()}`;
  },
  // idempotency key
  `${param1}:${param2}`);

  return out;
}

Example: Using more advanced options of the API

  • It will deactivate the hashing mechanism, so in the database you can see the actual contents of param1:param2 and there is no chance of collision.

  • Create function

import idempotender from '@idempotender/core';

const idem = idempotender({
  executionTTL: 24 * 3600,
  dynamoDBTableName: 'IdempotencyExecutions',
  keyHash: false,
});

function myIdempotentFunction(param1: string, param2: string): string {
  // get current idempotency status
  // acquire lock to avoid concurrent calls to this function
  const execution = await idem.getExecution(`${param1}:${param2}`);
  // already processed before, so return previous output data response
  if (execution.statusCompleted()) {
    return execution.output();
  }
  // another process is processing this in parallel
  if (execution.statusLocked()) {
    throw new Error(`Concurrent processing for idempotency key ${execution.key}`);
  }

  // status is "open", so
  // DO BUSINESS HERE
  try {
    console.log(`I only hello world for (${param1},${param2}) once!`);
    const output = `This is output '${param1}${param2}' at ${new Date()}`;
  } catch (err) {
    console.log('Error during function execution');
    idem.cancel(execution.key);
  }

  // save execution output so further calls in the next 24h
  // using the same "param1,param2" as key will return this response
  // without actually running the business function
  idem.complete(execution.key, output);

  return output;
}

Reference

  • These are the default values of the configuration
const idem = idempotender({
    dynamoDBTableName: 'IdempotencyExecutions',
    lockEnable: true,
    lockTTL: 10,
    lockRetryTimeout: 15,
    executionTTL: 24 * 3600,
    keyHash: true
}
  • Config attributes:

    • dynamoDBTableName

      • DynamoDB table to use for controlling idempotency

      • Defaults to 'IdempotencyExecutions'

    • lockEnable

      • Avoids parallel concurrency situations in which, while the first execution is running, another request arrives (before the first is finished)

      • In this situation, the second request will be responded with error 409 (conflict) so the client can resubmit it again later (and then receive a valid response, cached from the first run - because of the idempotency)

      • Activating this may increase your DynamoDB costs by ~3x, but is recommended because it's safer

      • Defaults to 'true'

    • lockTTL

      • Time in seconds that the concurrency lock will be active. During this timespan, if another request arrives, before the first request is finished, it will receive a status 419 (see lockEnable)

      • Defaults to '10'

    • lockRetryTimeout

      • Time in seconds waiting for an existing lock to be released in case of concurrency. If the lock is released in this period, we will try to get the last saved output from the other process (if it was saved) and can return the previous output gracefully. If after lock is released and output was not saved, we will try to acquire the lock again.

      • Defaults to '15'

    • executionTTL

      • Time in seconds after execution is finished in which if another request with the same key arrives, will be responded with the first execution response.

      • The actual execution will be skipped, but the client will receive the same response as in the first call.

      • Defaults to '24 * 3600'

    • keyHash

      • Whatever hash the key before storing in the database or not

      • Useful in cases where the key is too large or you don't want to expose the input parameters in plain text in DynamoDB

      • This is applied after keyMapper is executed

      • SHA-256 is used, which is very secure, but keep in mind that there is a very low possibility of collisions if this is enabled.

      • Defaults to 'true'

Functions

  • completeExecution(key:string, output:string)

    • Save the output as the execution result for a specific key, so that next time anyone tries to "getExecution(key)" with this key, it will be returned
  • getExecution(input:string):object

    • Queries the execution table for a certain input. If it doesn't exist and lock is enabled, creates the record for lock control

    • The actual key used in database operations is the result of the 'input' mapped to a key (see config 'keyMapper' and 'keyJmespath') and, if enabled, hashed with SHA-256.

    • Returns an object with struct:

      • statusCompleted():boolean

        • Returns true if a previous execution with corresponding output data was saved and is available via attribute 'output'
    • statusLocked():boolean

      • Returns true if another function started processing this but didn't reached "saveExecution()" still, indicating these two executions might be running in parallel, but only one should be able to actually run
    • output: previously saved output, indicating this function was already called before

  • deleteExecution(key:string)

    • Deletes an execution. Normally used when something goes wrong during the function execution and you want to clear the execution so another call can try to execute the function again later on