flipkart-scraper
v2.0.0
Published
[![NPM](https://nodei.co/npm/flipkart-scraper.png)](https://nodei.co/npm/flipkart-scraper/)
Downloads
29
Maintainers
Readme
Flipkart Scraper
This package will help us to scrape all Flipkart products through Flipkart affiliate API.
Please refer API Documentation here.
See the Examples here
Installation
Install using 'npm'
npm i flipkart-scraper
Install using 'yarn'
yarn add flipkart-scraper
Usage
import { FlipkartScraper } from "flipkart-scraper";
const scraper = new FlipkartScraper(
"<Affiliate-Id-Here>",
"<Affiliate-Token-Here>"
);
// 'data' event handler
scraper.on("data", (data) => {
console.log(data.products.length);
});
// Start the scraper
scraper.start();
Class: FlipkartScraper
- extends EventEmitter
This module will help us to scrape all the Flipkart products.
constructor(affiliateId, affiliateToken, [options])
This will create instance of the FlipkartScraper
class with required authentication params.
- options {object}
concurrency?
{number} Number for parallel processing in the queue, Default set to2
maxRequest?
{number} Maximum request to Flipkart affliate server, Default set to0
- means unlimitedmaxPage?
{number} Maximum number of pages to scrape per category, Default set to0
- means unlimited
Example
const scraper = new FlipkartScraper(
"<Affiliate-Id-Here>",
"<Affiliate-Token-Here>",
{
/**
* It will make 5 parallel request to Flipkart.
* This is optional param, default is set to 2
**/
concurrency: 5,
/**
* It will make only 500 request to Flipkart. After that program ends.
* This is optional param, default is set to 0 means unlimited
**/
maxRequest: 500,
/**
* Maximum 3 request per category
**/
maxPage: 3,
}
);
scraper.start([categoriesToScrape])
This method will start scraping through Flipkart affiliate API.
categoriesToScrape?
{string[]} Pass the list of categories that you want to scrape. Default set to[]
which means all categories.
Example
scraper.start(["telivision", "mobiles"]); // It will scrape only specified categories
scraper.stats(showAsNumbers?=false)
This method will show the stats of scraper. By default stats will come as numerals like (3.1k, 1.45GB)
showAsNumbers?
{boolean} Stats will return as number instead of numerals.
Sample Stats
/*
{
"startTime": "2021-02-21T08:14:29.445Z",
"endTime": undefined, // endTime will be available once scraping finished
"status": "inprogress",
"concurrency": 30,
"waitingRequests": 0,
"productsCount": "19.54k",
"elapsed": "0:00:06 10ms",
"durationPerMillionProducts": "0:05:08 558ms",
"productsPerSec": "3.25k products/sec",
"avgResponseTime": "147ms",
"requestPerSec": "7/sec",
"requestedCount": "43.00",
"processedCount": "41.00",
"errorCount": "0.0",
"retryCount": "0.0",
"retryHaltCount": "0.0",
"pendingCategory": 2,
"completedCategory": 1,
"downloadSize": "84.78MB",
"downloadedSpeed": "14.11MB/sec",
"info": {
"pendingCategories": [
{
"category": "mens_clothing",
"startTime": "2021-02-21T08:14:29.602Z",
"noOfPages": 18,
"elapsed": 5668,
"totalProducts": 9000
},
{
"category": "mobiles",
"startTime": "2021-02-21T08:14:29.603Z",
"noOfPages": 13,
"elapsed": 5771,
"totalProducts": 6500
}
],
"completedCategories": [
{
"category": "laptops",
"noOfPages": 10,
"totalProducts": 4500,
"elapsed": 3878
}
]
}
}
*/
events: 'response'
Emitted when successful HTTP response from the Flipkart Affiliate server.
Example
// 'response' event handler
scraper.on("response", (response) => {
console.log(response);
});
events: 'data'
Emitted when products returned from Flipkart affiliate API.
Example
// 'data' event handler
scraper.on("data", (data) => {
console.log(data.apiData.products);
});
events: 'categoryCompleted'
Emitted when all products scraped for the category.
Example
// 'categoryCompleted' event handler
scraper.on("categoryCompleted", (completedCategoryInfo) => {
console.log(completedCategoryInfo);
});
events: 'finished'
Emitted when scraper finished.
Example
// 'finished' event handler
scraper.on("finished", (info) => {
console.log(info);
});
events: 'error'
Emitted if any errors occured.
Example
// Triggered if any error occured
scraper.on("error", (error) => {
console.error(error);
});
events: 'retry'
Emitted if any retry occured.
Example
// Triggered if any retry occured
scraper.on("retry", (retryInfo) => {
console.log(retryInfo);
});
events: 'retryHalted'
Emitted if any retries failed for 10 times.
Example
// Triggerd when retry failed 10 times
scraper.on("retryHalted", (retryHaltInfo) => {
console.error(retryHaltInfo);
});