Newsagent

Monitor data sources, get alerted when they change.

Installing

$ npm install -g newsagent

Alternatively, don't install it and just prepend the command with npx.

Usage

$ newsagent <watchfiles...>

Where <watchfiles...> are names of one or more watchfiles.

Watchfiles

A watchfile defines how a data source should be monitored, what transforms to apply to those changes, and when to fire alerts. They are written in Yaml. An example:

name: AP Politics Twitter

source:
    method: fetch-html
    url: 'https://twitter.com/search?f=live&q=from:AP_Politics'
    browser: chromium
    selection: 'article div[lang="en"]'

schedule: 10s

monitor: additions-only

processes:
    - method: match-transform
      match: '/BREAKING: ([A-Za-z]+) ([A-Za-z ]+) wins (.+) to (.+) in (.+)\. #APracecall at (.+)\./'
      transform: '\\4: \\5 -- \\3 of \\2 (\\1) at \\6'
    - method: find-and-replace
      find: 'U.S.'
      replace: 'US'

alerts:
    - method: log

A watchfile can also have a top-level array with multiple watches.

The sections:

name

A unique name for this watchfile.

source

Where the data is going to come from. The method field should be one of the following source methods. The other source fields are specified by the method.

► fetch-html

Fetch a HTML page, and extract text from it.

• url The URL to fetch
• browser (optional) Specify either chromium, webkit, or firefox to fetch the page using that browser, if not specified simply fetches the HTML
• selection A CSS selector for one or more text elements, or the more advanced Playwright format if a browser is specified; to get the content of an attribute use a pipe (|) followed by the attribute name
• subselection (optional) Fields within that selection, each with their own selector

► fetch-json

Fetch a Json file, and extract data from it.

• url The URL to fetch
• selection A JmesPath selector

► fetch-csv

Fetch a CSV file.

• url The URL to fetch

schedule

How often to check for changes. Supports various human-readable formats.

monitor

What kinds of changes do you want to be alerted on? Set to either additions-and-removals, additions-only, or removals-only.

processes

An optional section. A list of processes which the changes are pushed through, one after another. They can modify or filter out what goes through to fire alerts.

► select

Select specific fields to retain from the data. Expects data input and outputs data.

• fields Specify fields, each with their own JmesPath selector

► filter

Filters out anything not matching a regular expression. Expects text/text-array input unless field is specified, and outputs the same.

• match A regular expression
• field (optional) The field within the input data to manipulate.

► trim

Remove whitespace at the start and end. Expects text/text-array input unless field is specified, and outputs the same.

• field (optional) The field within the input data to manipulate.

► split

Splits out new fields from text. Expects text/text-array input unless field is specified, and outputs the same.

• fields Specify fields to output, each with their own regular expression.
• field (optional) The field within the input data to extract data from

► combine

Combine different fields together. Expects data input and outputs text unless field is specified.

• combination The combined text, with fields included using double braces, eg. {{field-name}}
• field (optional) The name of a new output field.

► match-transform

Select parts of text using a regular expression and transform them. Expects text/text-array input and outputs text, unless field is specified.

• match A regular expression
• transform Text to output, using groups from the match, eg. \\1
• field (optional) The field within the input data to manipulate.

► find-and-replace

Looks for text and replaces it with something else. Expects text/text-array input unless field is specified, and outputs the same.

• find The text to look for.
• replace What to replace it with.
• field (optional) The field within the input data to manipulate.

alerts

A list of alerts to fire for all changes that have made it this far.

► log

Just prints to the console.

► email

Sends an email.

• to The email address to send to.
• smtpHost The host name of the SMTP server (for a Gmail account, use smtp.gmail.com)
• smtpUsername Your SMTP username (for a Gmail account, use your email address)
• smtpPassword Your SMTP password (for a Gmail account, use an app password)

► macos-notification

Shows a MacOS notification. Expects text input unless bodyField is specified.

• bodyField (optional) A specific field to use for the notification text.
• url (optional) A field containing a URL, which will be opened if the notification is clicked.

► webhook

Call a webhook, such as a Slack Webhook Workflow.

• url A webhook URL to call
• bodyField (optional) A specific field to use for the message text.

Inspiration

• Some unnamed tool built by the LA Times
• Workbench
• Pipedream
• Morph
• Huginn
• Easymorph
• Visualping, nee Change Detection
• Stakeout
• Datawire
• Datastringer
• Yahoo Pipes