puppeteer-extra

A light-weight wrapper around puppeteer to enable cool plugins through a clean interface.

Installation

yarn add puppeteer puppeteer-extra
# - or -
npm install puppeteer puppeteer-extra

You can also use a specific puppeteer version:

# puppeteer-extra works with any version, e.g. the prerelease:
yarn add puppeteer@next puppeteer-extra

Quickstart

// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality.
// Any number of plugins can be added through `puppeteer.use()`
const puppeteer = require("puppeteer-extra")

// Add stealth plugin and use defaults (all tricks to hide puppeteer usage)
const StealthPlugin = require("puppeteer-extra-plugin-stealth")
puppeteer.use(StealthPlugin())

// Add plugin to anonymize the User-Agent and signal Windows as platform
const UserAgentPlugin = require("puppeteer-extra-plugin-anonymize-ua")
puppeteer.use(UserAgentPlugin({ makeWindows: true }))

// That's it, the rest is puppeteer usage as normal 😊
puppeteer.launch({ headless: true }).then(async browser => {
  const page = await browser.newPage()
  await page.setViewport({ width: 800, height: 600 })

  console.log(`Testing the user agent plugin..`)
  await page.goto("https://httpbin.org/headers")
  await page.waitFor(1000)
  await page.screenshot({ path: "headers.png", fullPage: true })

  console.log(`Testing the stealth plugin..`)
  await page.goto("https://bot.sannysoft.com")
  await page.waitFor(5000)
  await page.screenshot({ path: "stealth.png", fullPage: true })

  console.log(`All done, check the screenshots. ✨`)
  await browser.close()
})

The above example uses the stealth and anonymize-ua plugin, which need to be installed as well:

yarn add puppeteer-extra-plugin-stealth puppeteer-extra-plugin-anonymize-ua
# - or -
npm install puppeteer-extra-plugin-stealth puppeteer-extra-plugin-anonymize-ua

If you'd like to see debug output just run your script like so:

DEBUG=puppeteer-extra,puppeteer-extra-plugin:* node myscript.js

Plugins

🆕 puppeteer-extra-plugin-recaptcha

• Solves reCAPTCHAs automatically, using a single line of code: page.solveRecaptchas().

🔥 puppeteer-extra-plugin-stealth

• Applies various evasion techniques to make detection of headless puppeteer harder.

puppeteer-extra-plugin-devtools

• Makes puppeteer browser debugging possible from anywhere.
• Creates a secure tunnel to make the devtools frontend (incl. screencasting) accessible from the public internet

puppeteer-extra-plugin-repl

• Makes quick puppeteer debugging and exploration fun with an interactive REPL.

puppeteer-extra-plugin-block-resources

• Blocks resources (images, media, css, etc.) in puppeteer.
• Supports all resource types, blocking can be toggled dynamically.

puppeteer-extra-plugin-flash

• Allows flash content to run on all sites without user interaction.

puppeteer-extra-plugin-anonymize-ua

• Anonymizes the user-agent on all pages.
• Supports dynamic replacing, so the browser version stays intact and recent.

puppeteer-extra-plugin-user-preferences

• Allows setting custom Chrome/Chromium user preferences.
• Has itself a plugin interface which is used by e.g. puppeteer-extra-plugin-font-size.

Check out the packages folder for more plugins.

Contributing

PRs and new plugins are welcome! :tada: The plugin API for puppeteer-extra is clean and fun to use. Have a look the PuppeteerExtraPlugin base class documentation to get going and check out the existing plugins (minimal example is the anonymize-ua plugin) for reference.

We use a monorepo powered by Lerna (and yarn workspaces), ava for testing, the standard style for linting and JSDoc heavily to auto-generate markdown documentation based on code. :-)

Kudos

• Thanks to skyiea for this PR that started the project idea.
• Thanks to transitive-bullshit for suggesting a modular plugin design, which was fun to implement.

Compatibility

puppeteer-extra and all plugins are tested continously against Node v9, v10 and Puppeteer v1.4 to v1.9 and @next, as well as a any combination thereof.

A few plugins won't work in headless mode (it's noted if that's the case) due to Chrome limitations (e.g. the user-preferences plugin), look into xvfb-run if you still require a headless experience in these circumstances.

API

Table of Contents

• PuppeteerExtra
  • use
  • launch
  • connect
  • plugins
  • getPluginData
  • executablePath
  • defaultArgs
  • createBrowserFetcher

PuppeteerExtra

Modular plugin framework to teach puppeteer new tricks.

This module acts a drop-in replacement for puppeteer.

Allows PuppeteerExtraPlugin's to register themselves and to extend puppeteer with additional functionality.

Type: function ()

Example:

const puppeteer = require("puppeteer-extra")
puppeteer.use(require("puppeteer-extra-plugin-anonymize-ua")())
puppeteer.use(
  require("puppeteer-extra-plugin-font-size")({ defaultFontSize: 18 })
)
;(async () => {
  const browser = await puppeteer.launch({ headless: false })
  const page = await browser.newPage()
  await page.goto("http://example.com", { waitUntil: "domcontentloaded" })
  await browser.close()
})()

use

Outside interface to register plugins.

Type: function (plugin): this

• plugin PuppeteerExtraPlugin

Example:

const puppeteer = require('puppeteer-extra')
puppeteer.use(require('puppeteer-extra-plugin-anonymize-ua')())
puppeteer.use(require('puppeteer-extra-plugin-user-preferences')())
const browser = await puppeteer.launch(...)

launch

Launch a new browser instance with given arguments.

Augments the original puppeteer.launch method with plugin lifecycle methods.

All registered plugins that have a beforeLaunch method will be called in sequence to potentially update the options Object before launching the browser.

Type: function (options): Puppeteer.Browser

• options Object? Regular Puppeteer launch options (optional, default {})

connect

Attach Puppeteer to an existing Chromium instance.

Augments the original puppeteer.connect method with plugin lifecycle methods.

All registered plugins that have a beforeConnect method will be called in sequence to potentially update the options Object before launching the browser.

Type: function (options)

• options {browserWSEndpoint: string, ignoreHTTPSErrors: boolean} (optional, default {})

plugins

Get all registered plugins.

Type: Array<PuppeteerExtraPlugin>

getPluginData

• See: puppeteer-extra-plugin/data

Collects the exposed data property of all registered plugins. Will be reduced/flattened to a single array.

Can be accessed by plugins that listed the dataFromPlugins requirement.

Implemented mainly for plugins that need data from other plugins (e.g. user-preferences).

Type: function (name)

• name string? Filter data by name property (optional, default null)

executablePath

Regular Puppeteer method that is being passed through.

Type: function (): string

defaultArgs

Regular Puppeteer method that is being passed through.

Type: function ()

createBrowserFetcher

Regular Puppeteer method that is being passed through.

Type: function (options): PuppeteerBrowserFetcher

• options Object?