callmeback

Serverless-friendly background task processing library. This library lets you enqueue tasks (represented as HTTP POST requests) to be processed in the background by utilizing cloud-based task queue services. Adapters are available for:

• Google Cloud Tasks
• Amazon SNS
• QStash by Upstash
• Zeplo
• In-process adapter (for local development and testing)

Synopsis

// Import the function and adapter you want to use.
import { callMeBack, GoogleCloudTasksAdapter } from 'callmeback'

// Import the SDK to use with the adapter.
import { CloudTasksClient } from '@google-cloud/tasks'

// Define the configuration.
const config = {
  adapter: new GoogleCloudTasksAdapter({
    client: new CloudTasksClient(),
    queuePath: 'projects/callmeback/locations/us-central1/queues/callmeback',
    url: 'https://.../process-background-task',
  }),
  // Other available adapters:
  // - AmazonSNSAdapter
  // - InProcessAdapter
  // - QStashAdapter
  // - ZeploAdapter
}

// Enqueue a background task.
const { id } = await callMeBack(config, {
  // Arbitrary JSON data to be passed to the background task.
  type: 'send-email',
  orderId: '1',
})

// The task ID is generated by the provider, can be used to e.g. search for logs.
console.log(`Task ID: ${id}`)

// Some time later, a POST request will be sent to the URL with
// the specified JSON data as the request body.

Design

Due to differences in the way each service works, this library makes the following trade-offs in order to make it extremely easy to switch to other providers when needed:

• The request method is always POST.
• The request body is always JSON-encoded. Some providers doesn’t allow setting request headers, so your endpoint should be configured to always decode JSON body, even if Content-Type is not application/json.
• The target URL is fixed. Many providers lets you set the target URL on a per-task basis, but some requires a target URL to be pre-configured (e.g. Amazon SNS). On providers that allows setting the URL for each task, the URL can be configured directly on the adapter (e.g. Google Cloud Tasks).
• Retry logic depends on the service. Amazon SNS has a default retry policy but it can be configured on a topic or subscription. On Google Cloud Tasks, a RetryConfig can be configured on a queue.
• Rate limiting (throttling) depends on the service. Amazon SNS lets you configure a throttle policy at the topic or subscription level. Google Cloud Tasks lets you configure RateLimits on a queue.
• Backpressure. The provider will call your endpoint as fast as it could (unless throttling is enabled). Your service should be configured to send a 504 (Gateway Timeout) response if it is overloaded. 429 (Too Many Requests) is not recommended because some providers will consider all 4xx responses as permanent failures and stop retrying.
• It is your service’s responsibility to verify the authenticity of incoming requests. An easy way is to embed some secret key when invoking callMeBack() and verify that the secret key is present on the way back. Or use some kind of signature or JWT.
• Due to retrying, your service may receive duplicate requests. It is your responsibility to make sure that your background job can be properly retried (e.g. by deduplicating requests or making sure actions are idempotent or conflict-free).
• Your service should respond in 10 seconds. Some providers will consider a request as failed if it takes too long to respond. If your service needs more time to process a request. If it will take more than 10 seconds, you should split the work into multiple tasks.

Usage with Google Cloud Tasks

About Google Cloud Tasks:

• The free tier provides 1,000,000 billable operations per month. Adding tasks to a queue and a delivery attempt each count as a billable operation. Therefore, 1 task consumes at least 2 operations, but may be more if retrying is needed. Billing is required and outgoing data transfer is charged separately.
• Allows the URL to be configured on a per-task basis.
• The URL does not need to be confirmed.
• You can configure the retry parameters on a queue.
• You can set up rate limiting on a queue. It allows setting the dispatch rate (in dispatches/second) and the number of maximum concurrent dispatches.
• Respects the Retry-After response header, and retries with a higher backoff rate if 429 (Too Many Requests) or 503 (Service Unavailable) is returned.
• Provides a dashboard for viewing metrics and ongoing task statuses.
• Logging can be turned on for individual tasks, but it is not enabled by default, as it may generate a large number of logs and can increase costs. Once enabled, it can be viewed in the Cloud Logging dashboard. Note that this is an all-or-nothing setting (no sampling).
• You can configure the concurrency limit to avoid overloading your application.
• Task timeout is 10 minutes. Although Google Cloud Tasks lets you configure the timeout up to 30 minutes, this library does not let you change it (see the design section for the reason).

Setting up:

1. Create a task queue.

   

2. Give it a name and region.

   

   Take note of the name, region ID, and project ID (found on the dashboard).

3. Construct a queue path using this format:

   projects/PROJECT_ID/locations/LOCATION_ID/queues/QUEUE_ID

4. Grant access to the queue:

   

Creating an adapter:

import { CloudTasksClient } from '@google-cloud/tasks'

const adapter = new GoogleCloudTasksAdapter({
  client: new CloudTasksClient(),
  queuePath: process.env.CALLMEBACK_CLOUD_TASK_QUEUE,
  url: 'https://.../',
})

Expected environment variables:

# PROJECT_ID is the ID of the Google Cloud project, found on the Google Cloud Console dashboard.
# LOCATION_ID is the ID of the location where the queue is located, e.g. "us-central1".
# QUEUE_ID is the ID of the queue, e.g. "callmeback".
CALLMEBACK_CLOUD_TASK_QUEUE=projects/PROJECT_ID/locations/LOCATION_ID/queues/QUEUE_ID

# When providing service account credentials to the SDK via environment variables.
# Note that there may be better ways to provide credentials to the SDK
# depending on your environment and use case.
# See: https://cloud.google.com/docs/authentication/provide-credentials-adc
GOOGLE_APPLICATION_CREDENTIALS=

Common errors:

• Error: Could not load the default credentials.
  • May be fixed by providing the credentials to the SDK.
• Error: 3 INVALID_ARGUMENT: Invalid resource field value in the request.
  • May be fixed by correctly configuring the queuePath (i.e. the CALLMEBACK_CLOUD_TASK_QUEUE environment variable) and making sure the url is valid.
• Error: 7 PERMISSION_DENIED: The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/…/locations/…/queues/…" (or the resource may not exist).
  • May be fixed by making sure the queue exists, the queuePath is correctly configured, permission to create tasks in the queue is granted to the user or service account (by using the “Cloud Tasks Enqueuer” role).

Monitor the metrics in Cloud Tasks dashboard:

Check the logs in Google Cloud Logging:

Inspecting the outstanding tasks in Cloud Tasks dashboard:

Usage with Amazon SNS

About Amazon SNS:

• The free tier provides 100,000 deliveries per month. Billing is required and outgoing data transfer is charged separately.
• You need to pre-configure the URL when creating the subscription.
• Once the subscription is created, you must confirm the subscription. Amazon SNS will send a request to the URL you configured. That request body will contain a SubscribeURL parameter. You must make a GET request to that URL to confirm the subscription. You can make your endpoint do that automatically, or you can take the URL and visit it manually in your browser.
• You can configure a retry policy and a throttle policy on the topic or subscription.
• I am not sure whether SNS respects the Retry-After response header or not. If someone was able to test this, please let us know and update the documentation.
• Amazon SNS provides metrics viewable in CloudWatch out-of-the-box. However, individual message delivery status logging must be configured and viewed in CloudWatch. It lets you adjust the sampling rate to save costs.
• Request timeout is 15 seconds (not configurable).
• Delivery is considered successful if the response status code is in the range of 200–4xx.

Setting up:

1. Create a topic:

   

2. Set it as a standard queue:

   

3. Take note of the topic ARN. Create a subscription.

   

4. Make it an HTTPS subscription and set an endpoint. Make sure to enable the “Raw message delivery” option.

   

5. Confirm the subscription by checking the SubscribeURL.

   

6. Grant permission to access the queue.

   

Creating an adapter:

import { SNS } from '@aws-sdk/client-sns'

const adapter = new AmazonSNSAdapter({
  topicArn: process.env.CALLMEBACK_SNS_TOPIC_ARN,
  sns: new SNS({}),
})

Expected environment variables:

CALLMEBACK_SNS_TOPIC_ARN=

# When providing credentials to the SDK via environment variables.
# Note that there may be better ways to provide credentials to the SDK
# depending on your environment and use case.
# See: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-iam.html
AWS_REGION=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

Common errors:

• Error: Region is missing
  • May be fixed by setting the AWS_REGION environment variable.
• CredentialsProviderError: Could not load credentials from any providers
  • May be fixed by providing the credentials to the SDK.
• AuthorizationErrorException: User: … is not authorized to perform: SNS:Publish on resource: … because no identity-based policy allows the SNS:Publish action
  • May be fixed by granting the user permission to publish to the topic.
• InvalidParameterException: Invalid parameter: TopicArn or TargetArn Reason: no value for required parameter
  • May be fixed by correctly configuring the topicArn (i.e. the CALLMEBACK_SNS_TOPIC_ARN environment variable).
• NotFoundException: Topic does not exist
  • May be fixed by creating the topic (and making sure the topic ARN is correctly configured).

Monitor metrics in CloudWatch:

Setting up logging:

Usage with QStash

About QStash:

• Very easy to configure.
• The free plan allows up to 500 requests per day (amounts to 14,000–15,500 requests per month) with maximum 3 retries.
• Retry count can be configured, but the back-off rate is fixed.
• Provides an easy-to-use dashboard for viewing tasks.

Setting up:

1. Copy the API key

   

Creating an adapter:

const adapter = new QStashAdapter({
  url: 'https://.../',
  token: process.env.QSTASH_TOKEN,
  retry: 3,
})

Expected environment variables:

QSTASH_TOKEN=

Inspecting requests in the dashboard:

Running tests

By default, the tests will only run against the in-process adapter.

To run the tests against other adapters, there are some preparation steps.

1. Run the requestbin script: node scripts/requestbin.mjs

   This script starts a web server that lets the tests verify the behavior.

2. Expose the requestbin service so that it’s publicly available.

   • If you are developing in Codespaces, you can go to the Ports tab, right click on the server’s Local Address → Port Visibility → Public.
   • You can also use ngrok or Cloudflare Quick Tunnel to expose the service.

   Then set the environment variable REQUESTBIN_URL to the URL of the requestbin service without the trailing slash, for example:

   export REQUESTBIN_URL=https://dtinth-verbose-computing-machine-rr4g7rgqv2jgj-35124.preview.app.github.dev

3. Set up the credentials for the other adapters.

   • Check out the above sections for more details.

Usage with Zeplo

About Zeplo:

• Very easy to configure.
• The free plan allows up to 500 requests per month.
• Flexible retry policies.
• Provides an easy-to-use dashboard for viewing tasks.
• Allows inspecting the response body and headers.
• Provides a CLI with the zeplo dev simulator that allows local development.

Setting up:

1. Copy the API key

   

Creating an adapter:

const adapter = new ZeploAdapter({
  zeploUrl: process.env.ZEPLO_URL,
  url: 'https://.../',
  token: process.env.ZEPLO_TOKEN,
  retry: 3,
})

Expected environment variables:

ZEPLO_TOKEN=

# If you want to test locally with `zeplo dev`, uncomment the next line:
ZEPLO_URL=http://localhost:4747

Inspecting requests in the dashboard: