hacker-news-favorites-api

A simple script that will scrape the favorites for a provided hacker news user id.

Before ⇨ After 👍

Why?

The official Hacker News API from Y Combinator doesn't include the favorites endpoint. I use favorites like bookmarks and wanted to sync them to another app.

Usage

You can run yourself, or use the api endpoint hosted at the below url. Be sure to pass the query params mentioned below.

Base Endpoint:

https://reactual.api.stdlib.com/hnfavs/

Example URL:

This example would download 1 page of 30 favorites for the user sbr464

https://reactual.api.stdlib.com/hnfavs/?id=sbr464&limit=1

Accepts 3 query params, only id is required.

id

required | string | default='' | The Hacker News username.

limit

optional | number | default=1 | Max # of pages to download (30/page).

offset

optional | number | default=1 (none) | Pagination offset, the page to begin on.

Data Format

The API will return a JSON object-array containg the favorites as below. You can use the article id to obtain more info via the Hacker News API. There is more info (comments link, score) avail to scrape from favorites, but the HN API is more reliable.

;[
  {
    // HN article id
    id: '17437185',

    // Article's direct url (not comments)
    link: 'https://github.com/serhii-londar/open-source-mac-os-apps',

    // Title of the article on HN
    title: 'Awesome macOS open source applications'
  }
]

Notes

Currently, the request timeout is manually set to just under 3 minutes, but could change. You may need to provide an offset and make multiple requests if you are getting a timeout when trying to download hundreds-thousands+ of favorites. If you receive a timeout error, modify the limit/offset and retry the request. We are using stdlib to host the api.

More Info

Additional docs & a live testing tool avail at stdlib.

Installation via npm

Recently published as hnfavs on npm, have only done basic testing. x-ray is a required peer dependency, you'll need to add to your project.

npm install --save hnfavs x-ray

yarn add hnfavs x-ray

// Basic usage
const hnfavs = require('hnfavs')

hnfavs('sbr464').then(console.log)

Local Installation

It's a node.js app, using x-ray for the web scraping. See src/main.test.js for example usage locally.

# Clone repo
$ git clone [email protected]:reactual/hacker-news-favorites-api.git

$ cd hacker-news-favorites-api

# Install with yarn or npm
$ yarn

# Run tests
$ yarn test