npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

scraperjs

v1.2.0

Published

A complete and versatile web scraper.

Downloads

1,513

Readme

Scraperjs

Build Status Dependency Status Coverage Status NPM version Inline docs

Scraperjs is a web scraper module that make scraping the web an easy job.

Installing

npm install scraperjs

If you would like to test (this is optional and requires the installation with the --save-dev tag),

grunt test

To use some features you’ll need to install phantomjs, if you haven’t already

Getting started

Scraperjs exposes two different scrapers,

  • a SimpleScraper, that is light fast and with a low footprint, however it doesn't allow for more complex situations, like scraping dynamic content.
  • a DynamicScraper, that is a bit more heavy, but allows you to scrape dynamic content, like in the browser console. both scrapers expose a very similar API, with some minor differences when it comes to scraping.

Lets scrape Hacker News, with both scrapers.

Try to spot the differences.

Static Scraper

var scraperjs = require('scraperjs');
scraperjs.StaticScraper.create('https://news.ycombinator.com/')
	.scrape(function($) {
		return $(".title a").map(function() {
			return $(this).text();
		}).get();
	})
	.then(function(news) {
		console.log(news);
	})

The scrape promise receives a function that will scrape the page and return the result, it only receives jQuery a parameter to scrape the page. Still, very powerful. It uses cheerio to do the magic behind the scenes.

Dynamic Scraper

var scraperjs = require('scraperjs');
scraperjs.DynamicScraper.create('https://news.ycombinator.com/')
	.scrape(function($) {
		return $(".title a").map(function() {
			return $(this).text();
		}).get();
	})
	.then(function(news) {
		console.log(news);
	})

Again, the scrape promise receives a function to scrape the page, the only difference is that, because we're using a dynamic scraper, the scraping function is sandboxed only with the page scope, so no closures! This means that in this (and only in this) scraper you can't call a function that has not been defined inside the scraping function. Also, the result of the scraping function must be JSON-serializable. We use phantom and phantomjs to make it happen, we also inject jQuery for you.

However, it's possible to pass JSON-serializable data to any scraper.

The $ varible received by the scraping function is, only for the dynamic scraper, hardcoded.

Show me the way! (aka Routes)

For a more flexible scraping and crawling of the web sometimes we need to go through multiple web sites and we don't want map every possible url format. For that scraperjs provides the Router class.

Example

var scraperjs = require('scraperjs'),
	router = new scraperjs.Router();

router
	.otherwise(function(url) {
	console.log("Url '"+url+"' couldn't be routed.");
});

var path = {};

router.on('https?://(www.)?youtube.com/watch/:id')
	.createStatic()
	.scrape(function($) {
		return $("a").map(function() {
			return $(this).attr("href");
		}).get();
	})
	.then(function(links, utils) {
		path[utils.params.id] = links
	})

router.route("https://www.youtube.com/watch/YE7VzlLtp-4", function() {
	console.log("i'm done");
});

Code that allows for parameters in paths is from the project Routes.js, information about the path formating is there too.

API overview

Scraperjs uses promises whenever possible.

StaticScraper, DynamicScraper and ScraperPromise

So, the scrapers should be used with the ScraperPromise. By creating a scraper

var scraperPromise = scraperjs.StaticScraper.create() // or DynamicScraper

The following promises can be made over it, they all return a scraper promise,

  • onStatusCode(code:number, callback:function(utils:Object)), executes the callback when the status code is equal to the code,
  • onStatusCode(callback:function(code:number, utils:Object)), executes the callback when receives the status code. The callback receives the current status code,
  • delay(time:number, callback:function(last:?, utils:Object)), delays the execution of the chain by time (in milliseconds),
  • timeout(time:number, callback:function(last:?, utils:Object)), executes the callback function after time (in milliseconds),
  • then(lastResult:?, callback:function(last:?, utils:Object)), executes the callback after the last promise,
  • async(callback:function(last:?, done:function(result:?, err:?), utils)), executes the callback, stopping the promise chain, resuming it when the done function is called. You can provide a result to be passed down the promise chain, or an error to trigger the catch promise,
  • catch(callback:function(error:Error, utils:Object)), executes the callback when there was an error, errors block the execution of the chain even if the promise was not defined,
  • done(callback:function(last:?, utils:Object)), executes the callback at the end of the promise chain, this is always executed, even if there was an error,
  • get(url:string), makes a simple HTTP GET request to the url. This promise should be used only once per scraper.
  • request(options:Object), makes a (possibly) more complex HTTP request, scraperjs uses the request module, and this method is a simple wrapper of request.request(). This promise should be used only once per scraper.
  • scrape(scrapeFn:function(...?), callback:function(result:?, utils:Object)=, ...?), scrapes the page. It executes the scrapeFn and passes it's result to the callback. When using the StaticScraper, the scrapeFn receives a jQuery function that is used to scrape the page. When using the DynamicScraper, the scrapeFn doesn't receive anything and can only return a JSON-serializable type. Optionally an arbitrary number of arguments can be passed to the scraping function. The callback may be omitted, if so, the result of the scraping may be accessed with the then promise or utils.lastReturn in the next promise.

All callback functions receive as their last parameter a utils object, with it the parameters of an url from a router can be accessed. Also the chain can be stopped.

DynamicScraper.create()
	.get("http://news.ycombinator.com")
	.then(function(_, utils) {
		utils.stop();
		// utils.params.paramName
	});

The promise chain is fired with the same sequence it was declared, with the exception of the promises get and request that fire the chain when they've received a valid response, and the promises done and catch, which were explained above.

You can also waterfall values between promises by returning them (with the exception of the promise timeout, that will always return undefined) and it can be access through utils.lastReturn.

The utils object

You've seen the utils object that is passed to promises, it provides useful information and methods to your promises. Here's what you can do with it:

  • .lastResult, value returned in the last promise
  • .stop(), function to stop the promise chain,
  • .url, url provided to do the scraper,
  • .params, object with the parameters defined in the router matching pattern.
A more powerful DynamicScraper.

When lots of instances of DynamicScraper are needed, it's creation gets really heavy on resources and takes a lot of time. To make this more lighter you can use a factory, that will create only one PhantomJS instance, and every DynamicScraper will request a page to work with. To use it you must start the factory before any DynamicSrcaper is created, scraperjs.DynamicScraper.startFactory() and then close the factory after the execution of your program, scraperjs.DynamicScraper.closeFactory(). To make the scraping function more robust you can inject code into the page,

var ds = scraperjs.DynamicScraper
	.create('http://news.ycombinator.com')
	.async(function(_, done, utils) {
		utils.scraper.inject(__dirname+'/path/to/code.js', function(err) {
			// in this case if there was an error won't fire catch promise.
			if(err) {
				done(err);
			} else {
				done();
			}
		});
	})
	.scrape(function() {
		return functionInTheCodeInjected();
	})
	.then(function(result) {
		console.log(result);
	});

Router

The router should be initialized like a class

var router = new scraperjs.Router(options);

The options object is optional, and these are the options:

  • firstMatch, a boolean, if true the routing will stop once the first path is matched, the default is false.

The following promises can be made over it,

  • on(path:string|RegExp|function(url:string)), makes the promise for the match url or regular expression, alternatively you can use a function to accept or not a passed url. The promises get or request and createStatic or createDynamic are expected after the on promise.
  • get(), makes so that the page matched will be requested with a simple HTTP request,
  • request(options:Object), makes so that the page matched will be requested with a possible more complex HTTP request, , scraperjs uses the request module, and this method is a simple wrapper of request.request(),
  • createStatic(), associates a static scraper to use to scrape the matched page, this returns ScraperPromise, so any promise made from now on will be made over a ScraperPromise of a StaticScraper. Also the done promise of the scraper will not be available.
  • createDynamic(), associates a dynamic scraper to use to scrape the matched page, this returns ScraperPromise, so any promise made from now on will be made over a ScraperPromise of a DynamicScraper. Also the done promise of the scraper will not be available.
  • route(url:string, callback:function(boolean)), routes an url through all matched paths, calls the callback when it's executed, true is passed if the route was successful, false otherwise.
  • use(scraperInstance:ScraperPromise), uses a ScraperPromise already instantiated.
  • otherwise(callback:function(url:string)), executes the callback function if the routing url didn't match any path.
  • catch(callback:function(url:string, error:Error)), executes the callback when an error occurred on the routing scope, not on any scraper, for that situations you should use the catch promise of the scraper.

Notes

  • Scraperjs always fetches the document with request, and then when using a DynamicScraper, leverages phantom's setContent() to set the body of the page object. This will result in subtly different processing of web pages compared to directly loading a URL in PhantomJS.

More

Check the examples, the tests or just dig into the code, it's well documented and it's simple to understand.

Dependencies

As mentioned above, scraperjs is uses some dependencies to do the the heavy work, such as

  • async, for flow control
  • request, to make HTTP requests, again, if you want more complex requests see it's documentation
  • phantom + phantomjs, phantom is an awesome module that links node to phantom, used in the DynamicScraper
  • cheerio, light and fast DOM manipulation, used to implement the StaticScraper
  • jquery, to include jquery in the DynamicScraper
  • although Routes.js is great, scraperjs doesn't use it to maintain it's "interface layout", but the code to transform the path given on the on promise to regular expressions is from them

License

This project is under the MIT license.