Webscraper Exporter

A simple yet powerful prometheus exporter for website performance metrics built using puppeteer.

Table of contents

• Webscraper Exporter
  • Table of contents
  • CLI
    • Getting started
      • Installation of the CLI
      • Configuration
      • Using the CLI
        • Starting
        • Creating a project
        • Adding, Removing and Listing Templates
  • NodeJS module
    • Installation of the module
    • Using the module
  • Docker
  • Exported Data
  • Customization
  • Contributing

CLI

Currently only tested on macOS and Linux, should support Windows

Getting started

Installation of the CLI

You can install the cli included to get started quickly with barely no configuration:

npm install --global webscraper-exporter 

or

yarn global add webscraper-exporter

You can then check if everything went OK by running:

wsce --version

This should print the package's version

Configuration

In your current working directory, create a file named wsce.config.js with the following example content:

module.exports = {
    scraper: {
        urls: ["https://google.com"],
        puppeteerOptions: {},
        addons: [],
        interval: 60_000,
    },
    exporter: {
        port: 9924
    },
};

For further information on the configuration, see CONFIGURING.md

Using the CLI

To see all available options for a command, see wsce <command> -h;

Starting

You can start the exporter by simply executing:

wsce start

You can then access the metrics at http://localhost:9924

If you want more detailled logs, you can pass the -v argument.

To pass a custom config file, use -c path/to/file

Creating a project

Create a project from one of the included templates by running:

wsce init my-project 

You will then be prompted for the project template. You can also directly pass the template to wsce init, e.g. :

wsce init my-project --template javascript

By default, wsce includes a typings file for the config so it's easier to fill. If you don't want that, you can pass --typings=false to the init command.

Adding, Removing and Listing Templates

wsce template has 3 subcommands:

• add: Add a template from a local directory / remote repository
• remove: Remove a saved template
• list: List all saved templates

For more information on templates, see TEMPLATES.md

NodeJS module

Installation of the module

npm install --save webscraper-exporter 

or

yarn add webscraper-exporter

Using the module

import { Exporter, Scraper } from "webscraper-exporter";

const scraper = new Scraper({
    urls: ["https://cstef.dev"],
    addons: [
        {
            name: "Logger",
            when: "before",
            run: (browser, page, URL) => {
                console.log(`I am running on ${URL}`);
            },
        },
    ],
    interval: 60_000,
});
scraper.start();
const exporter = new Exporter({ 
    scraper, 
    port: 9924, 
});
exporter.start();

For more examples, see the examples folder.

Docker

There is a Dockerfile included in this repository so you can build your own image for wsce (based on the node:alpine-16 image).

See DOCKER.md for more information.

Exported Data

You can see the metrics that are exposed in: EXPORTED_DATA.md

Customization

You can easily write your own plugins in Javascript to expand webscraper-exporter's functionalities.

See ADDONS.md for more information.

Contributing

Pull requests are always welcome if you feel like fixing / expanding something in this project !