goose-parser

This tool moves routine crawling process to the new level. Now it's possible to parse a web page for a few moments. All you need is to specify parsing rules based on css selectors. It's so simple as Goose can do it. This library allows to parse such data types as grids, collections, and simple objects. Parser supports pagination via infinite scroll and pages. It offers next features: pre-parse actions and post-parse transformations.

Key features

• Declarative approach for definition of parsing rules, actions and transformations.
• Multi environments to run parser on the browser and server sides.
• Clear and consistent API with promises all the way.
• Improved Sizzle format of selectors.
• Ajax and multi-pages parsing modes.
• Easy extendable.

Table of contents:

• Installation
• Documentation
  • Environments
    • PhantomEnvironment
    • BrowserEnvironment
  • Parser
    • #parse method
    • #addAction Custom action
    • #addTransformation Custom transformation
    • #addPagination Custom pagination
  • Parse rules
    • Simple rule
    • Collection rule
    • Grid rule
  • Pagination
    • Scroll pagination
    • Page pagination
  • Actions
    • Click
    • Wait
    • Type
    • Exist
    • ConditionalActions
    • Custom actions
  • Transformations
    • Date
    • Replace
    • Custom transformations
• Tests
  • With PhantomEnvironment
  • With BrowserEnvironment
• Debug
• Usage

Installation

npm install goose-parser

This library has dependency on PhantomJS 2.0. Follow instructions provided by the link or build it manually.

Documentation

All css selectors can be set in a sizzle format.

Environments

This is a special atmosphere where Parser has to be executed. The main purpose of the environment is to provide a method for evaluating JS on the page.

PhantomEnvironment

That environment is used for running Parser on node.

var env = new PhantomEnvironment({
    url: 'http://google.com',
});

The main and only required parameter is url. It contains an url address of the site, where Parser will start.

This environment allows to perform snapshots, use proxy lists, custom proxy rotator, white and black lists for loading resources and more sweet features. Find more info about options in here.

BrowserEnvironment

That environment is used for running Parser in the browser.

var env = new BrowserEnvironment();

To created packed js-file with Parser execute following command:

npm run build

Parser

Parser.js is the main component of the package which performs page parsing.

var parser = new Parser({
    environment: env,
    pagination: pagination
});

Fields:

• environment - one of the allowed environments.
• pagination [optional] - pagination definition.

#parse method

parser.parse({
    actions: actions,
    rules: parsingRules
});

Fields:

• actions [optional] - Array of actions to execute before parsing process.
• rules - parsing rules which define scopes on the page.

#addAction Custom action

Add custom action by using method addAction. Custom function is aware about context of Actions.

Example

parser.addAction('custom-click', function(options) {
    // do some action
});

Params:

• type - name of the action.
• action - function to execute when action is called.

#addTransformation Custom transformation

Add custom trasformation by using method addTransformation.

Example

parser.addTransformation('custom-transform', function (options, result) {
    return result + options.increment;
});

Params:

• type - name of the transformation.
• transformation - function to execute when transformation is called.

#addPagination Custom pagination

Add custom pagination by using method addPagination. Custom pagination is aware about context of Paginator.

Usage

parser.addPagination('custom-pagination', function (options) {
    // Paginate function
    // return vow.resolve();
}, function (options, timeout) {
    // Check pagination function
    // return vow.resolve();
});

Params:

• type - name of new pagination method.
• paginateFn - Function performs pagination, should return Promise.
• checkPaginationFn - Function checks pagination complete, should return Promise.

Example

Describe new pagination type

var previousPageHtml;
parser.addPagination('clickPerPage', function (options) {
    var selector = options.scope + ':eq(' + this._currentPage + ')';
    return this._env
        .evaluateJs(options.pageScope, this._getPaginatePageHtml)
        .then(function (html) {
            previousPageHtml = html;
            return this._actions.click(selector);
        }, this);
}, function (options, timeout) {
    return this._actions.wait(this._getPaginatePageHtml, function (html) {
        return html !== null && html !== previousPageHtml;
    }, [options.pageScope], timeout)
});

Use it

var pagination = {
    type: 'clickPerPage', // your custom type
    pageScope: '.page__content',
    scope: '.page__pagination'
};

var parser = new Parser({
    environment: env,
    pagination: pagination
});

parser.parse({
    rules: {}
});

Parse rules

Simple rule

The purpose of this rule - retrieving simple textual node value(s).

Example:

Parsing rule

{
    name: 'node',
    scope: 'div.simple-node'
}

HTML

<div class='simple-node'>simple-value</div>

Parsing result

{
    node: 'simple-value'
}

Fields:

• name - name of the node which is presented in the result dataSet.
• scope - css selector of the node.
• id [optional] - true|Function. Will mark a row as identifier, the result will be set to node with _id name. If function specified, parser will call it for each row. See more info in example.
• attr [optional] - name of attribute to parse. If specified, parser will parse node.getAttribute(attr)
• separator [optional] - separator applies to glue the nodes after parse, if nodes more than one.
• type [optional] - (array|string[default]). Allows to specify result data type.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• actions [optional] - see Actions.
• transform [optional] - see Transformations.

Collection rule

The purpose of this rule - retrieving collection of nodes.

Example:

Parsing rule

{
    name: 'row',
    scope: 'div.collection-node',
    collection: [
        {
            name: 'node1',
            scope: 'div.simple-node1'
        },
        {
            name: 'node2',
            scope: 'div.simple-node2'
        },
        {
            name: 'nested',
            scope: 'div.nested-node',
            collection: [
                {
                    name: 'node3',
                    scope: 'div.simple-node3'
                }
            ]
        }
    ]
}

HTML

<div class='collection-node'>
    <div class='simple-node1'>simple-value1</div>
    <div class='simple-node2'>simple-value2</div>
    <div class='nested-node'>
        <div class='simple-node3'>simple-value3</div>
    </div>
</div>

Parsing result

{
    row: {
        node1: 'simple-value1',
        node2: 'simple-value2',
        nested: {
            node3: 'simple-value3'
        }
    }
}

Fields:

• name - name of the node which is presented in the result dataSet.
• scope - css selector of the node.
• collection - array of any rule types.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• actions [optional] - see Actions.
• transform [optional] - see Transformations.

Grid rule

The purpose of this rule - retrieving collection of collection.

Example:

Parsing rule

{
    scope: 'div.collection-node',
    collection: [[
        {
            name: 'node1',
            scope: 'div.simple-node1'
        },
        {
            name: 'node2',
            scope: 'div.simple-node2'
        }
    ]]
}

HTML

<div>
    <div class='collection-node'>
        <div class='simple-node1'>simple-value1</div>
        <div class='simple-node2'>simple-value2</div>
    </div>
    <div class='collection-node'>
        <div class='simple-node1'>simple-value3</div>
        <div class='simple-node2'>simple-value4</div>
    </div>
</div>

Parsing result

[
    {
        node1: 'simple-value1',
        node2: 'simple-value2'
    },
    {
        node1: 'simple-value3',
        node2: 'simple-value4'
    }
]

Fields:

• scope - css selector of the node.
• collection - array of array of any rule types.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• actions [optional] - see Actions.
• transform [optional] - see Transformations.

Examples with predefined row id

Parsing rule with id = function

var id = 0;
{
    scope: 'div.collection-node',
    collection: [[
        {
            id: function (rule, result) {
               return ++id;
            },
            scope: 'simple-reference'
        },
        {
            name: 'node',
            scope: 'div.simple-node'
        }
    ]]
}

Parsing rule with id from scope

{
    scope: 'div.collection-node',
    collection: [[
        {
            id: true,
            scope: 'simple-reference'
        },
        {
            name: 'node',
            scope: 'div.simple-node'
        }
    ]]
}

HTML

<div>
    <div class='collection-node'>
        <div class='simple-reference'>1</div>
        <div class='simple-node'>simple-value1</div>
    </div>
    <div class='collection-node'>
        <div class='simple-reference'>2</div>
        <div class='simple-node'>simple-value2</div>
    </div>
</div>

Parsing result

[
    {
        _id: 1,
        node2: 'simple-value1'
    },
    {
        _id: 2,
        node2: 'simple-value2'
    }
]

Pagination

This is a way to parse collection-based data. See more info in Paginator.js

Scroll pagination

This type of pagination allows to parse collections with infinite scroll.

{
    type: 'scroll',
    interval: 500
}

Fields:

• type - "scroll" for that type of pagination.
• interval - interval in pixels to scroll.
• maxPagesCount [optional] - max pages to parse.
• maxResultsCount [optional] - max results count.
• timeout [optional] - timeout for paginate action.

Page pagination

This type of pagination allows to parse collections with ajax-page pagination.

JS definition

{
    type: 'page',
    scope: '.page',
    pageScope: '.pageContainer',
}

HTML

<div>
    <div class='pageContainer'>
        <div class='collection-node'>
            <div class='simple-node1'>simple-value1</div>
            <div class='simple-node2'>simple-value2</div>
        </div>
        <div class='collection-node'>
            <div class='simple-node1'>simple-value3</div>
            <div class='simple-node2'>simple-value4</div>
        </div>
    </div>
    <div class='pagination'>
        <div class='page'>1</div>
        <div class='page'>2</div>
        <div class='page'>3</div>
    </div>
</div>

Fields:

• type - "page" for that type of pagination.
• scope - css selector for paginator block (page label).
• pageScope - css selector for page scope (container for page-data).
• strategy - [optional] ajax|newPage - pagination strategy, if empty will be detected automatically.
• maxPagesCount [optional] - max pages to parse.
• maxResultsCount [optional] - max results count.
• timeout [optional] - timeout for paginate action.

Actions

Allow to execute actions on the page before parse process. All actions could return a result of the execution.

Click

Click by the element on the page.

Example:

{
    type: 'click',
    scope: '.open-button'
}

Fields:

• type - click for that action.
• scope - css selector of the node.
• waitForPage [optional] - true|false. Wait for the page reload, could be useful when click handles page refresh.
• waitForPageTimeout [optional] - timeout for waitForPage action.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• once [optional] - to perform action only once (can be useful on pre-parse moment).

Wait

Wait for the element on the page.

Example:

{
    type: 'wait',
    scope: '.open-button.done'
}

Fields:

• type - wait for that action.
• scope - css selector of the node.
• timeout [optional] - time to cancel wait in seconds.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• once [optional] - to perform action only once (can be useful on pre-parse moment).

Type

Type text to the element.

Example:

{
    type: 'type',
    scope: 'input'
    text: 'Some text to enter'
}

Fields:

• type - type for that action.
• scope - css selector of the node.
• text - text to enter to the element.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).

Exist

Check if element exist on the page.

Example:

{
    type: 'exist',
    scope: '.some-element'
}

Fields:

• type - exist for that action.
• scope - css selector of the node.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).

ConditionalActions

Action which helps to create if statement based on another action.

Example:

{
    type: 'conditionalActions',
    conditions: [
        {
            type: 'exist',
            scope: '.element-to-check'
        }
    ],
    actions: [
        {
            type: 'click',
            scope: '.element-to-check',
            waitForPage: true
        }
    ]
}

In this particular action parser checks if element .element-to-check presents on the page, do action click on it.

Fields:

• type - conditionalActions for that action.
• conditions - Actions to check condition.
• actions - Actions which will be executed if conditions are true.

Custom actions

Add custom action by using method addAction. Custom function is aware about context of Actions.

Example

actions.addAction('custom-click', function(options) {
    // do some action
});

Transformations

Allow to transform parsed value to some specific form.

Date

Format date to specific view (using momentjs).

{
    type: 'date',
    locale: 'ru',
    from: 'HH:mm D MMM YYYY',
    to: 'YYYY-MM-DD'
}

Replace

Replace value using Regex.

{
    type: 'replace',
    re: ['\\s', 'g'],
    to: ''
}

Custom transformations

Add custom trasformation by using method addTransformation.

Example

transformations.addTransformation('custom-transform', function (options, result) {
    return result + options.increment;
});

Tests

With PhantomEnvironment

To run tests use command:

npm test

With BrowserEnvironment

To run tests build them with command:

npm run build-test

And then run file in the browser.

Debug

All parser components are covered by debug library, which give an ability to debug application in easy way. Set DEBUG variable with name of js file to show debug information.

DEBUG=Parser,Actions app.js

Usage

var env = new PhantomEnvironment({
    url: uri,
    screen: {
        width: 1080,
        height: 200
    }
});

var parser = new Parser({
    environment: env,
    pagination: {
        type: 'scroll',
        interval: 500
    }
});

parser.parse({
    actions: [
        {
            type: 'wait',
            timeout: 2 * 60 * 1000,
            scope: '.container',
            parentScope: 'body'
        }
    ],
    rules: {
        scope: '.outer-wrap',
        collection: [[
            {
                name: 'node1',
                scope: '.node1',
                actions: [
                    {
                        type: 'click',
                        scope: '.prepare-node1'
                    },
                    {
                        type: 'wait',
                        scope: '.prepare-node1.clicked'
                    }
                ],
                collection: [
                    {
                        name: 'subNode',
                        scope: '.sub-node',
                        collection: [[
                            {
                                name: 'date',
                                scope: '.date-node',
                                transform: [
                                    {
                                        type: 'date',
                                        locale: 'ru',
                                        from: 'HH:mm D MMM YYYY',
                                        to: 'YYYY-MM-DD'
                                    }
                                ]
                            },
                            {
                                name: 'number',
                                scope: '.number-node'
                            }
                        ]]
                    }
                ]
            },
            {
                name: 'prices',
                scope: '.price'
            }
        ]]
    }
}).done(function (results) {
    // do whatever with results
});