ScrapingAnt API client for JavaScript

@scrapingant/scrapingant-client is the official library to access ScrapingAnt API from your JavaScript applications. It runs both in Node.js and browser and provides useful features like automatic retries and parameters encoding to improve the ScrapingAnt usage experience.

• Quick Start
• API key
• Retries with exponential backoff
• API Reference
• Examples

Quick Start

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the example.com site.
client.scrape('https://example.com')
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

API key

In order to get API key you'll need to register at ScrapingAnt Service

Retries with exponential backoff

Network communication sometimes fails, that's a given. The client will automatically retry requests that failed due to a network error, an internal error of the ScrapingAnt API (HTTP 500+). By default, it will retry up to 8 times. First retry will be attempted after ~500ms, second after ~1000ms and so on. You can configure those parameters using the maxRetries and minDelayBetweenRetriesMillis options of the ScrapingAntClient constructor.

API Reference

All public classes, methods and their parameters can be inspected in this API reference.

ScrapingAntClient

ScrapingAntClient is the official library to access ScrapingAnt API from your JavaScript applications. It runs both in Node.js and browser.

• ScrapingAntClient
  • new ScrapingAntClient(options)
  • .scrape(url, [params]) ⇒ ScrapingAnt API response

new ScrapingAntClient(options)

| Param | Type | Default | |----------------------------------------|---------------------|------------------| | [options] | object | | | [options.maxRetries] | number | 8 | | [options.minDelayBetweenRetriesMillis] | number | 500 | | [options.timeoutSecs] | number | 60 | | [options.apiKey] | string | |

scrapingAntClient.scrape(url, [parameters]) ⇒ ScrapingAnt API response

https://docs.scrapingant.com/request-response-format#available-parameters

| Param | Type | |--------------------------------|----------------------| | url | string | | [parameters] | object | | [parameters.browser] | boolean | | [parameters.cookies] | string | | [parameters.headers] | object | | [parameters.js_snippet] | string | | [parameters.proxy_type] | string | | [parameters.proxy_country] | string | | [parameters.wait_for_selector] | string | | [parameters.return_text] | boolean |

IMPORTANT NOTE: parameters.js_snippet will be encoded to Base64 automatically by the ScrapingAnt JS client library.

ScrapingAntApiError

An ScrapingAntApiError is thrown for successful HTTP requests that reach the API, but the API responds with an error response. Typically, those are internal errors, which are automatically retried, or validation errors, which are thrown immediately, because a correction by the user is needed.

Properties

| Name | Type | Description | |------------|---------------------|------------------------------------| | message | string | Error message returned by the API. | | statusCode | number | HTTP status code of the error. | | httpMethod | string | HTTP method of the API call. |

Examples

Using residential proxy

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Get the residential IP info using httpbin.org
client.scrape('https://httpbin.org/ip', { proxy_type: 'residential' })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Sending custom cookies

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the httpbin.org site and get all the cookies sent before
client.scrape('https://httpbin.org/cookies', { cookies: 'cookieName1=cookieVal1;cookieName2=cookieVal2' })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Adding custom headers

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the httpbin.org site and get all the headers that would be sent before
client.scrape('https://httpbin.org/headers', { headers: { scraping: "is cool!" } })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));

Executing custom JS snippet

const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the httpbin.org site and replace all the content with "Hello, world"
const customJsSnippet = "var str = 'Hello, world!';\n" +
    "var htmlElement = document.getElementsByTagName('html')[0];\n" +
    "htmlElement.innerHTML = str;"
client.scrape('https://httpbin.org/cookies', { js_snippet: customJsSnippet })
    .then(res => console.log(res))
    .catch(err => console.error(err.message));