crawler-request

HTTP request module customized for crawlers.

Installation

npm install crawler-request

Usage

Simple Request

const crawler = require("crawler-request");

crawler("https://stackoverflow.com/").then(function (response) {
	// handle response
	console.log(response.text.lenght);
});

PDF Parse

const crawler = require("crawler-request");

crawler("http://careers.stackoverflow.com/stack_overflow_careers.pdf").then(
	function (response) {
		// handle response
		console.log(response.text.lenght);
	}
);

Extend

const crawler = require("crawler-request");

function response_text_size(response) {
	response["size"] = response.text.length;
	return response;
}

crawler("https://stackoverflow.com/", response_text_size).then(function (
	response
) {
	// handle response
	console.log(response.size);
});

Test

mocha or npm test

Check test folder and quickstart.js for extra usages.

Support

I use this package actively myself, so it has my top priority. You can chat on WhatsApp about any infos, ideas and suggestions.

Submitting an Issue

If you find a bug or a mistake, you can help by submitting an issue to GitLab Repository

License

MIT licensed and all it's dependencies are MIT or BSD licensed.