webrobber

v0.2.0

Published

2 years ago

A light weight nodejs library to helps you grab the specified content on any web page on internet.

Downloads

0High
0Medium
0Low

tanker327

webrobber web crawler content

WebRobber

WebRobber is a light weight nodejs library. Main goal of this library is to helps you grab the specified content on any web page on internet.

Example

Let's use http://www.google.com as a example. If you check the source code of the page , it should like below:

<html>
    <head>
        <title>google</title>
    </head>
    <body>
    ....
        <img src="/images/icons/product/chrome-48.png"/>
    ....
        <div>
            <img id="hplogo" src="'/images/srpr/logo9w.png'"/>
        </div>
    ....
    </body>
</html>

Now let's use WebRoober to grab the title of this page , url of the logo image and all the image urls.

var webRobber = require('webrobber');

var url = "http://www.google.com";
var dataMaps = {
        title:"title",
        logoImg:"#hplogo | src",
        images:"img | src"
        };

webRobber.grab(url, dataMaps, function (err, result) {
    if(err){
        console.log("Grab fail.");
    }else{
        console.log(result);
    }
});

The result will be :


{
  title: 'Google',
  logoImg: '/images/srpr/logo9w.png',
  images:  [
             '/images/icons/product/chrome-48.png',
             '/images/srpr/logo9w.png'
           ],
  _url: 'http://www.google.com'
}

To begin

Please make sure you have nodejs and npm installed.

Install it:
```
$ npm install webrobber --save
```
Require it:
```
var webRobber = require('webrobber');
```

Prepare dataMap:

var dataMap = { property name : selector [| attribute]}

example:

var dataMap = {
			    title:"title",
			    logoImg:"#hplogo | src"
   			};

Please see DataMap for details

Grab the content from web page

webRobber.grab(url, dataMap, function (err, result) {
	if(err){
   		console.log("Grab fail.");
	}else{
    	console.log(result);
	}
});

Or if you familiar with Promise (Q). You can do following:

webRobber.grab(url, dataMap)
    .then(function(result){
        console.log(result);
    })
    .catch(function(error){
        console.log(error);
    });

DataMap

DataMap is a map object use to define Name and Selector of the content you want to grap. Name is used be mark the name of content in the final reslut object. Selector is used to select the content in the web page.

WebRobber's selector usage is nearly identical to Cheerio's and jQuery's. Please check Cheerio's selector

The default grabed value will be the .text() of a dom selected by selector. If you need grab attribute of a dom. Please add " | attribute name".

Example: we want to grab the src of a image dom

var dataMap = {
				logoImg:"#hplogo | src"
       	};

If you may have several possible selectors for one content. You can put all possible selectors in an array. All the selector will be checked one after another until we find valid content.

var dataMap = {
				logoImg:["#hplogo | src","#logo-url","#new-logo | src"]
       	};

If multiple contents are found for one selector. All contents will be set an array and saved in result. Please see example at top for you reference.

If use * as selector, all the content of web page will be save for the property.

Result

All the results of grabbing will be return in a object as result (Please see example at top for detail).

There is special value in result object : _url. It keeps the url used for grabbing.

Test

To run the tests for WebRobber, please run:

$ npm test

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme