Fetch until complete

A TypeScript utility function for fetching paginated data until all data is retrieved or an optional timeout is exceeded.

Features

• Automatic Pagination: Continues fetching data until there are no more items.
• Timeout Control: Supports customizable timeout to prevent long-running operations.
• Logging Options: Provides options to enable or disable logs for cleaner output.
• Performance Tracking: Logs the time taken for the fetch operation when enabled.

Installation

You can install this utility via npm:

npm i pipedrive-fetch-until-complete

Usage

import {
  fetchUntil,
  FetchUntilTimeoutError,
  type AdditionalData,
} from 'pipedrive-fetch-until-complete';

async function fetchData(params) {
  // Your data fetching logic here.
  // Must return an object with `data` and optional `additionalData.pagination`.
}

const allData = await fetchUntil(
  'YourFetchOperationName',
  fetchData,
  { start: 0, limit: 100 },
  { timeout: 60000, hideLogs: false } // optional
);

Parameters

• name (string): Name of the fetch operation for logging purposes.
• fetchData (function): Function that fetches data given pagination parameters { start, limit }. It should return a promise resolving to an object { data: T[]; additionalData?: AdditionalData }.
• initialParams (object): Initial pagination parameters.
  • start (number): Starting index for pagination.
  • limit (number): Number of items to fetch per page.
• options (object, optional):
  • timeout (number): Maximum time in milliseconds to wait for the fetch operation to complete. Default is 60000 ms.
  • hideLogs (boolean): Whether to hide logs for this fetch operation. Default is false.

Return Value

• Promise<T[]>: Resolves to an array containing all fetched data.

Error Handling

• Throws FetchUntilTimeoutError if the fetch operation exceeds the specified timeout.

Error Handling

• Throws FetchUntilTimeoutError if the fetch operation exceeds the specified timeout.

Examples

Basic Example

import { fetchUntil } from './utils.js';

async function fetchData({ start, limit }) {
  // Simulate an API call
  const response = await apiCall({ start, limit });
  return {
    data: response.items,
    additionalData: response.additionalData,
  };
}

const allData = await fetchUntil('DataFetch', fetchData, {
  start: 0,
  limit: 500,
});

Handling Timeouts

import { fetchUntil } from './utils.js';

try {
  const allData = await fetchUntil(
    'DataFetch',
    fetchData,
    { start: 0, limit: 50 },
    { timeout: 5000 }, // 5 seconds timeout
  );
} catch (error) {
  if (error instanceof FetchUntilTimeoutError) {
    console.error('Fetch operation timed out.');
  } else {
    throw error;
  }
}

License

This project is licensed under the MIT License.