openscraping

v0.3.1

Published

2 years ago

Turn unstructured HTML pages into structured data. The OpenScraping library can extract information from HTML pages using a JSON config file with xPath rules. It can scrape even multi-level complex objects such as tables and forum posts.

Downloads

0High
0Medium
0Low

zmarty

OpenScraping HTML Structured Data Extraction Node.js library

This is the Node.js version. A separate but similar C# library is located here.

Self-contained example

First install the package using npm in your project:

npm install openscraping

Then paste this simple example in a js file and run it with node:

var openscraping = require('openscraping')

var config = `
{
  "title": "//h1",
  "body": "//div[contains(@class, 'article')]"
}
`

var html = '<html><body><h1>Article title</h1><div class="article">Article contents</div></body></html>'

var scrapingResults = openscraping.parse(JSON.parse(config), html)

console.log('Extracted title: ' + scrapingResults.title)
console.log('Extracted body: ' + scrapingResults.body)
console.log('Full extracted json: ' + JSON.stringify(scrapingResults))

Here is the output:

Extracted title: Article title
Extracted body: Article contents
Full extracted json: {"title":"Article title","body":"Article contents"}

OpenScraping API Server

If you want to directly run an API server with both a test console UI and an HTTP API, please take a look at the OpenScraping API Server. The API server does not contain a crawler, it just runs rules against HTML sent in with an HTTP POST.

Example: Extracting an article from bbc.com

Below is a simple configuration file that extracts an article from a www.bbc.com page.

{
  "title": "//div[contains(@class, 'story-body')]//h1",
  "dateTime": "//div[contains(@class, 'story-body')]//div[contains(@class, 'date')]",
  "body": "//div[@property='articleBody']"
}

Here is how to call the library:

// config contains the JSON config from above, html contains the HTML we want to extract data from
var openscraping = require('openscraping')
scrapingResults = openscraping.parse(JSON.parse(config), html)
console.log(scrapingResults)

And here is the result for a bbc news article:

{
  title: 'Robert Downey Jr pardoned for 20-year-old drug conviction',
  dateTime: '24 December 2015',
  body: 'Body of the article is shown here'
}

Here is how the www.bbc.com page looked like on the day we saved the HTML for this sample:

Example: Extracting a list of products from Ikea

The sample configuration below is more complex as it demonstrates support for extracting multiple items at the same time, and running transformations on them. For this example we are using a products page from ikea.com.

{
  "products": 
  {
    "_xpath": "//div[@id='productLists']//div[starts-with(@id, 'item_')]",
    "title": ".//div[contains(@class, 'productTitle')]",
    "description": ".//div[contains(@class, 'productDesp')]",
    "price": 
    {
      "_xpath": ".//div[contains(@class, 'price')]/text()[1]",
      "_mapTransformations": [
        "TrimTransformation"
      ]
    }
  }
}

Here is a snippet of the result:

{
  products: [{
    title: 'HEMNES',
    description: 'coffee table',
    price: '$139.00'
  },
...
  {
    title: 'NORDEN',
    description: 'sideboard',
    price: '$149.00'
  },
  {
    title: 'SANDHAUG',
    description: 'tray table',
    price: '$79.99'
  }]
}

Here is how the www.ikea.com page looked like on the day we saved the HTML for this sample:

Map and Reduce Transformations

In the Ikea example above we used a map transformation called TrimTransformation. Transformation modify the raw extracted HTML nodes in some ways. For instance, TrimTransformation just runs str.trim() on the extracted text before it gets written to the JSON output.

The difference between map and reduce transformations is that map transformations act on individual items (for instance on all paragraphs that match the rule //p), while reduce transformations act on the array of extracted items (in our case on all paragraphs). Reduce transformations can be used, for instance, to merge all extracted paragraphs into a single continuous string, because they receive an array of strings as input, and they can choose to return a single string as output.

Built-in map transformations

Name | Purpose | Example -------------------------------------------- | ------- | -------------- ParseDateTransformation | Uses the Date.parse function to parse a string into a date, then converts it back to a string with a certain date format. | Here RemoveExtraWhitespaceTransformation | Replaces consecutive spaces with a single space. For the string "hello world" it would return "hello world". | TrimTransformation | Runs str.trim() on the extracted text before it gets written to the JSON output. | Here TextExtractionBetterWhitespaceTransformation | The default text extractor just calls node.textContent, which often concatenates strings without adding white space between them. This implementation tries to solve this problem by adding extra white spaces in some cases. |

Built-in reduce transformations

Name | Purpose | Example -------------------------------------------- | ------- | -------------- MergeTextArrayIntoSingleText | Expects an array of strings as input, and outputs a single string that concatenates all strings from the input array. Each string is trimmed. If an array item is empty, it is ignored. The transformation concatenates strings using a single space character. | Here

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme