hardened-fetch

Hardened Fetch is a tiny wrapper for global.fetch adding request timeouts, throttling, rate limit awareness, retries, and descriptive errors. It makes working with APIs without SDKs and building robust web scrapers easier.

import { HardenedFetch } from 'hardened-fetch'

const client = new HardenedFetch({
  baseUrl: 'https://swapi.dev/api/',
  maxRetries: 3,
})

const response = await client.fetch('/species/1')

Installation

This is a Node.js module available through the npm registry. Node.js 20 or higher is required.

$ npm install hardened-fetch

Features

• Throttle concurrent requests and control request rate
• Retries failed requests with exponential back off
• Retries rate-limited requests according to reset time
• Supports request timeouts and multiple abort controllers

API

new HardenedFetch([options] = {})

Creates a new Hardened Fetch client.

Constructor Options:

| Name | Type | Description | | ------------------- | ------------- | -------------------------------------------------------------------------------------------------------- | | baseUrl | string | A base URL to prepend to each request, optional. | | defaultHeaders | HeadersInit | Default headers to add to each request, optional. | | maxInProgress | number | The maximum number of requests to execute per time interval. | | maxPerInterval | number | The maximum number of concurrent requests in progress. | | intervalLength | number | The length of each time interval in milliseconds. | | maxRetries | number | Number of retry attempts for failed requests. | | doNotRetryMethods | string[] | List of HTTP methods that will not trigger a retry attempt. | | doNotRetryCodes | number[] | List of HTTP status codes that will not trigger a retry attempt. | | rateLimitHeaders | string[] | The names of the rate limit reset headers to lookup. |

All of the options and their defaults are shown below:

const client = new HardenedFetch({
  // Request options
  baseUrl: undefined,
  defaultHeaders: undefined,
  // Throttle options
  maxInProgress: 20,
  maxPerInterval: 10,
  intervalLength: 1000,
  // Retry options
  maxRetries: 3,
  doNotRetryCodes: [400, 401, 403, 404, 405, 406, 410, 411, 412, 422, 451, 501],
  doNotRetryMethods: ['CONNECT', 'DELETE', 'PATCH', 'POST', 'PUT'],
  // Rate limit options
  rateLimitHeaders: ['Retry-After', 'RateLimit-Reset', 'X-RateLimit-Reset', 'X-Rate-Limit-Reset'],
})

client.fetch(url, [init={}], [timeout=30_000])

Expects a url to the resource that you wish to fetch, optionally settings to apply to the request and a timeout in milliseconds. Returns a promise which will resolve with the Response object when successful or rejects with an HTTPError for non-20x responses.

const response = await client.fetch('https://swapi.dev/api/species/1/')
const json = await response.json()

License

This package is MIT licensed.