node-crawler

Fast asynchronous NodeJS module for crawling/scraping a web through worker_threads.

Features

• Crawling on threads(CPU cores)
• Crawl images and links
• Configurable max threshold crawl
• Configurable retries
• Configurable request and retry delays
• Rotate proxies
• Bypass robots.txt

Demo: https://node-crawler-server.glitch.me

Get started

Install

npm i @abilashinamdar/node-crawler

    OR

yarn add @abilashinamdar/node-crawler

Basic usage

import crawl from "@abilashinamdar/node-crawler";

crawl(config, onEveryCrawl, onCrawlError, onCrawlComplete)

config

Key | Type | Default | Value --- | --- | --- | --- url | String | - | crawl | Array | ["links"] | ["links", "images"] maxCrawl | Number | 0 | Acts as threshold value. Stops crawling once it reaches the maxCrawl. 0 represents no threshold. maxRetries | Number | 0 | No. of times the request should be retried when fails. allowExternalImages| Boolean | false | true/false. true allows crawler to pick image if it points to outside the origin/host retryDelay| Number | 0 | in milliseconds. Delay between the retry requests. useProxy| Boolean | false | true/false. true allows crawler to rotate some free proxies from https://www.free-proxy-list.com proxies| Array | [] | Crawler will rotate the proxies from the list rather than fetching free proxies. useProxy: true is mandatory. skipRobotsFile| Boolean | false | true/false. true allows crawler to bypass the robots.txt check. requestDelay| Number | 0 | in milliseconds. Delay between the concurrent requests.

onEveryCrawl(value)

onEveryCrawl will get triggered when crawler will crawl Link or Image.
It will be triggered with the value as object which is as follows:
{
    type: string,
    link: string,
    reason: string
}

type - It can be 'LINK' / 'IMAGE' / 'IGNORE'
reason - Description for the IGNORE type.

onCrawlError(error)

onCrawlError will get triggered when crawler get error while crawling.
It will be triggered with the error as sample object as below or JS Error object.

{
    type: string,
    link: string,
    reason: string
}

type - It will be 'DISALLOWED'
reason - Description for the DISALLOWED type. Basically occurs when robots.txt disallows the url.

onCrawlComplete(value)

onCrawlComplete will get triggered when crawler done with the crawling.
It will be triggered with the value as object which is as follows:

{
    links: Set<String>,
    images: Set<String>,
    ignored: Set<String>
}

links - List of links from the crawled url.
images - List of image links from the crawled url.
ignored - List of ignored links from the crawled url.

Note: After this trigger the thread will get exit and will be available for others.

License

ISC