electron-microscope

Use electron to inspect websites and extract data. Intended for automation, testing, web scraping, etc.

Loads urls inside an electron webview tag, allows you to execute code on them, and stream data from the pages back to your main process.

Run this headlessly on Linux using xvfb-run.

BETA DISCLAIMER early adopters only, this module is still hecka fresh

usage

Use this in an electron app:

var electron = require('electron')
var createMicroscope = require('electron-microscope')

electron.app.on('ready', function () {
  createMicroscope(function (err, scope) {
    if (err) throw err
    // use your new microscope
  })
}) 

Run it with electron:

$ npm install electron-prebuilt -g
$ electron my-code.js

API

require('electron-microscope')(options, ready)

Requiring the module returns a constructor function that you use to create a new instance. Pass it an options object and a ready callback that will be called with (error, scope). scope is your new instance all ready to go.

scope.loadURL(url, cb)

Load a url, and call cb with (err) when loading is done. If there was a problem loading the page err will be the error, otherwise it means it loaded successfully

scope.run(code)

Run code the currently loaded page. Run this after calling loadURL. Code must be a string, if it is a function then .toString() will be called on it. scope.run returns a readable stream that emits data generated by your code.

You code must be a function that has this template:

function (send, done) {
  // put your custom code here
  // call 'send(data)' to write data to the stream
  // call 'done()' to end the stream
  // calling send is optional, but you must eventually call done
}

For example:

var scraper = `function (send, done) {
  for (var i = 0; i < 5; i++) send(i)
  done()
}`
var output = scope.run(scraper)
output.pipe(concat(function (out) {
  console.log(out.toString()) // 12345
}))

scope.on('will-navigate', cb)

Emitted the page wants to start navigation. It can happen when the window.location object is changed or a link is clicked in the page.

Calls cb with (url), forwarded from this event.

scope.on('did-start-loading', cb)

Corresponds to the points in time when the spinner of the tab starts spinning.

Calls cb with no arguments, forwarded from this event.

scope.on('did-stop-loading', cb)

Corresponds to the points in time when the spinner of the tab stops spinning.

Calls cb with no arguments, forwarded from this event.

scope.destroy()

Call when you don't want to use the scope anymore. Causes the browser-window elecron-microscope uses internally to close.