Unfurl

A metadata scraper with support for oEmbed, Twitter Cards and Open Graph Protocol for Node.js (>=v8.0.0)

The what

Unfurl (spread out from a furled state) will take a url and some options, fetch the url, extract the metadata we care about and format the result in a saine way. It supports all major metadata providers and expanding it to work for any others should be trivial.

The why

So you know when you link to something on Slack, or Facebook, or Twitter - they typically show a preview of the link. To do so they have crawled the linked website for metadata and enriched the link by providing more context about it. Which usually entails grabbing its title, description and image/player embed.

The how

npm install unfurl.js

unfurl(url [, opts])

url - string

opts - object of:

• oembed?: boolean - support retrieving oembed metadata
• timeout? number - req/res timeout in ms, it resets on redirect. 0 to disable (OS limit applies)
• follow?: number - maximum redirect count. 0 to not follow redirect
• compress?: boolean - support gzip/deflate content encoding
• size?: number - maximum response body size in bytes. 0 to disable
• userAgent?: string - User-Agent string is often used for content negotiation

import { unfurl } from 'unfurl.js'
const result = unfurl('https://github.com/trending')

result is <Promise<Metadata>>

type Metadata = {
  title?: string
  description?: string
  keywords?: string[]
  favicon?: string
  author?: string
  oEmbed?: {
    type: 'photo' | 'video' | 'link' | 'rich'
    version?: string
    title?: string
    author_name?: string
    author_url?: string
    provider_name?: string
    provider_url?: string
    cache_age?: number
    thumbnails?: [{
      url?: string
      width?: number
      height?: number
    }]
  }
  twitter_card: {
    card: string
    site?: string
    creator?: string
    creator_id?: string
    title?: string
    description?: string
    players?: {
      url: string
      stream?: string
      height?: number
      width?: number
    }[]
    apps: {
      iphone: {
        id: string
        name: string
        url: string
      }
      ipad: {
        id: string
        name: string
        url: string
      }
      googleplay: {
        id: string
        name: string
        url: string
      }
    }
    images: {
      url: string
      alt: string
    }[]
  }
  open_graph: {
    title: string
    type: string
    images?: {
      url: string
      secure_url?: string
      type: string
      width: number
      height: number
    }[]
    url?: string
    audio?: {
      url: string
      secure_url?: string
      type: string
    }[]
    description?: string
    determiner?: string
    site_name?: string
    locale: string
    locale_alt: string
    videos: {
      url: string
      stream?: string
      height?: number
      width?: number
      tags?: string[]
    }[]
    article: {
      published_time?: string
      modified_time?: string
      expiration_time?: string
      author?: string
      section?: string
      tags?: string[]
    }
  }
}

The who 💖

(If you use unfurl.js too feel free to add your project)

• vapid/vapid - A template-driven content management system
• beeman/micro-unfurl - small microservice that unfurls a URL and returns the OpenGraph meta data.
• probot/unfurl - a GitHub App built with probot that unfurls links on Issues and Pull Request discussions