Botler

API

Botler is simple and flexible scraping framework

Getting Started

Installation

To use Botler in your project, run:

yarn add botler
# or "npm i --save botler"

Note: When you install Puppeteer, it downloads a recent version of Chromium (~71Mb Mac, ~90Mb Linux, ~110Mb Win) that is guaranteed to work with the API. To skip the download, see Environment variables.

Usage

Caution: Botler requires at least Node v6.4.0, but the examples below use async/await which is only supported in Node v7.6.0 or greater.

import Joi from 'joi';
import botler from './src/index';

async () => {
  await botler.launch();

  const task = botler.createTask({
    type: botler.MINING_TASK_TYPE,
    description: 'Get page title',
    schema: Joi.object().keys({
      title: Joi.string().required(),
    }),
    async worker(page) {
      return {
        title: await page.title(),
      };
    },
  });

  const routine = botler.createRoutine([task]);
  const bot = await botler.createBot(routine);
  await bot.spawn('http://books.toscrape.com/');
})();