npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

flightplan-tool

v0.3.51

Published

Search for award inventory using Node.js and Headless Chrome!

Downloads

85

Readme

Flightplan

Flightplan is a Javascript library that makes it easy to scrape and parse airline websites for award inventory. It uses Puppeteer for scraping, which is built on top of Headless Chrome, meaning it behaves just like a full-fledged web browser, but it can be run from the command line with no visible window (allowing you to use your computer for other things). Furthermore, it can run on any platform supported by Headless Chrome, which is just about everything (Windows, Mac, or Linux).

Flightplan Web UI

Disclaimer: Scraping is generally against an airline's website's terms of service. As mentioned above, Flightplan typically doesn't place more load on a website than a normal human would, but unlike the human, it can run 24/7. So please use responsibly! Use of any scraping tool (or even excessive non-automated usage) can cause an airline to temporarily (or permanently) ban your IP or member account.

Website

Don't feel comfortable with a command line? Wish there was a way to use Flightplan without installing anything or waiting hours to collect data? Now you can! Just visit:

https://www.flightplantool.com

Supported Airlines

Airline | Website | Search | Parse | Notes ------------------------|-----------------------|:------------------:|:------------------:|:------ AC (Air Canada) | Aeroplan | :white_check_mark: | :white_check_mark: | BA (British Airways) | Executive Club | :white_check_mark: | :white_check_mark: | Award mileage not available CX (Cathay Pacific) | AsiaMiles | :white_check_mark: | :white_check_mark: | KE (Korean Air) | SKYPASS | :white_check_mark: | :white_check_mark: | Award mileage not available for partner awards NH (All Nippon Airways) | ANA Mileage Club | :white_check_mark: | :white_check_mark: | QF (Qantas) | Frequent Flyer | :construction: | :construction: | In progress SQ (Singapore Airlines) | KrisFlyer | :white_check_mark: | :white_check_mark: |

Installation

If you are a developer using Flightplan as a library in your own Javascript project:

$ npm install flightplan-tool
# or "yarn add flightplan-tool"

Otherwise, if you just want to run the Flightplan tools as an end-user, you can install globally:

$ npm install --global flightplan-tool
# or "yarn global add flightplan-tool"

Don't have npm installed? Read the setup guide

Usage

Not a developer? Check out the section on Flightplan's Command Line Interface.

With Flightplan, you specify an airline and get back an engine, which supports two operations: searching and parsing.

  1. Searching simply takes a query, and fetches the HTML response (or multiple responses for some websites, which break the results up across multiple tabs).
  2. Parsing takes those HTML responses, and returns the list of flight awards. (Note: currently, Flightplan ignores non-direct routes and partner awards, this may change in the future.)

This is useful, because searching is expensive, but parsing is cheap. So it makes sense to search once, but be able to parse many times (perhaps due to bug fixes or new features being added).

const fp = require('flightplan');

const cx = fp.new('cx');

(async () => {
  // Must call initialize before searching (replace credentials with real ones below)
  await cx.initialize({ credentials: ['1234567890', 'passw0rd'] });

  // Do a one-way search
  const { responses, error } = await cx.search({
    fromCity: 'HKG', toCity: 'LHR',
    departDate: '2019-03-06', cabin: 'first'
  });
  
  // Check for an error
  if (error) {
    console.log(error);
    return;
  }
  
  // Parse out awards from the responses
  const { awards } = cx.parse(responses);
  console.log(awards);
})();

You can also instruct the search engine to save both the HTML output, and even screenshots! :tada: This makes debugging what might've gone wrong later much easier. Let's try it out:

const fp = require('flightplan');

const sq = fp.new('sq');

(async () => {
  await sq.initialize({ credentials: ['1234567890', '123456'] });
  const { html, screenshots, error } = await sq.search({
    fromCity: 'SIN', toCity: 'HKG',
    departDate: '2019-09-06', cabin: 'business',
    html: { path: 'output.html' }, screenshot: { path: 'output.jpg' }
  });
    
  if (!error) {
    console.log('HTML Files:', html.length);
    console.log('Screenshots:', screenshots.length);
  }
})();

API

API Documentation

Command Line Interface

Flightplan comes bundled with a set of command line tools and web UI, that makes it easy to search for flight awards, without knowing how to code! If you're a more visual person, please check out the tutorial on YouTube:

http://www.youtube.com/watch?feature=player_embedded&v=QMtiucIPOxs

Note: The YouTube tutorial covers v1, so some things will be a bit different.

If you installed Flightplan globally (using the --global flag with npm) you can run the Flightplan command from anywhere. For a list of commands, run:

$ flightplan --help
Usage: flightplan [options] [command]

Options:
  -V, --version  output the version number
  -h, --help     output usage information

Commands:
  search         Search for award inventory.
  parse          Parse search results from database.
  import         Import search results from another directory.
  cleanup        Remove incomplete requests or data resources.
  stats          Print statistics on routes stored in the database.
  client         Launch the web client.
  server         Launch the web server.
  help [cmd]     display help for [cmd]

Each of the commands are covered in more detail below.

Working Directory

Whenever you run the flightplan command, your current working directory is important. Flightplan stores data in several subdirectories inside the working directory, and will create them automatically if it cannot find them. The subdirectories it looks for are:

config/ # Stores your account credentials in accounts.txt
data/   # Webpages / screenshots when searching awards are saved here
db/     # The database containing all requests and awards found

To keep things organized, it's recommended to create a directory just for Flightplan (for example C:\flightplan or ~/flightplan, although you can call it whatever you like). When running Flightplan, be sure to always run it from the same directory.

Commands

Search

Search a specific engine for awards, over a specified date range. If no query parameters are provided, they can be entered interactively on the command line. By default, Chrome will be visible, but can be hidden with the --headless flag. Awards will also be parsed and added to the database by default, but this can also be turned off with --no-parser. The search command is smart enough to avoid re-running queries that are already in the database, though this can be overridden with --force.

Example

# search AsiaMiles for JFK-HKG first-class awards in September 2019
$ flightplan search -w CX -f JFK -t HKG -c first -s 2019-09-01 -e 2019-09-30 -q 1

Parse

By default, when searching for awards, the HTML or JSON is parsed for awards which are then written to the database. Sometimes, due to bugs or updates to the parser code, it is necessary to re-parse the awards (using the HTML/JSON files saved to disk, instead of re-running all searches, which could be very time-consuming). The parse command will look for all requests, which have not yet been parsed (in case the --no-parser flag was used) and will parse them. If you wish to re-parse all requests, even those that have already been parsed, use the --force flag.

Example

# forcibly re-parse all AsiaMiles awards
$ flightplan parse -w CX --force

Import

If you are running Flightplan on one or more computers, and wish to combine multiple databases into a single one (so you can see all the results in the Web UI), the import command is used to do this. It takes the path of a database, which it will add to the database in the current working directory. In the case of conflicts (e.g. the same search exists in both databases) the data for the more recent one will be used. Request resources (HTML/JSON files and screenshots) will also be imported.

Example

# import data from another directory
$ flightplan import -d ~/Downloads/my-other-flightplan

Note: May need some fixes to work properly in V2.

Cleanup

Normally, there is a one-to-one mapping between requests in the database, and resource files saved in the data subdirectory. If either become orphaned they can be cleaned up using this command. You can also get rid of old data (by providing an age), so you can get more recent data when running the search command (otherwise the older data will prevent the searches from being re-run unless the --force flag is used).

Example

# remove all requests older than 2 months (durations are in ISO 8601 format)
$ flightplan cleanup -m P2M

Note: May need some fixes to work properly in V2.

Stats

Prints the # of requests and awards stored in the database, for all unique routes (grouped by cabin and quantity). Useful to know which routes have already been searched, and which may need additional searching to be complete.

$ flightplan stats
Opening database...
Analyzing requests table...
Analyzing awards table...

BKK-SYD:
  NH [first, 1x]: 2 requests, 9 awards
SYD-BKK:
  NH [first, 1x]: 2 requests, 9 awards
SIN-MEL:
  SQ [first, 2x]: 8 requests, 80 awards
MEL-SIN:
  SQ [first, 2x]: 5 requests, 49 awards
ORD-LHR:
  BA [economy, 1x]: 35 requests, 488 awards
LHR-ORD:
  BA [economy, 1x]: 11 requests, 134 awards

Totals:
  6 routes (3 unique)
  49 requests
  769 awards

Client

Runs the Web UI, which can be accessed by pointing your web browser at http://localhost:3000.

Important: You must also run the API server, so the client can fetch award data (see the next command).

Server

Runs the back-end server, which reads the database and provides an API endpoint for the web client to query available awards.