npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

hred

v1.5.1

Published

![hred](./.github/hred.svg)

Downloads

5

Readme

hred

hred (html reduce) is a command-line tool to extract data from HTML. It reads HTML from the standard input and outputs the JSON produced by a qsx query:

> curl https://danburzo.ro/rolodex/ | hred "article a { @href, @.textContent }"
[
  {
    "href": "http://www.3quarksdaily.com/",
    ".textContent": "3 Quarks Daily"
  },
  {
    "href": "http://50watts.com",
    ".textContent": "50 Watts"
  },
  {
    "href": "http://aworkinglibrary.com/",
    ".textContent": "A Working Library"
  },
  ...
]

The qsx documentation describes the kinds of queries you can make with hred, but if you're familiar with CSS selectors you're mostly good to go.

Installation

hred runs on Node.js. You can find hred in the npm registry:

# install hred globally with npm:
npm install -g hred

# install hred globally with yarn:
yarn global add hred

# run hred without installing it:
npx hred 

Usage

hred accepts a qsx query string:

curl https://en.wikipedia.org/wiki/Banana | hred "img { @alt, @src }"

[
  {
    "alt": "Page semi-protected",
    "src": "//upload.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/20px-Semi-protection-shackle.svg.png"
  },
  {
    "alt": "Banana and cross section.jpg",
    "src": "//upload.wikimedia.org/wikipedia/commons/thumb/f/ff/Banana_and_cross_section.jpg/250px-Banana_and_cross_section.jpg"
  },
  ...
]

hred has the single, modest purpose of extracting parts of HTML as JSON. Because the qsx query language is a lightweight extension to the CSS selector syntax used by the Element.querySelectorAll() DOM method, hred offers only limited reshaping of the resulting JSON via aliases. The tool is designed to be piped to something like jq if further JSON processing is necessary.

hred has a few options available:

Option | Description ------ | ----------- -c, --concat | if the result is an array, return it as concatenated JSON records, to make it easier to collate several results together -f <queryfile>, --file=<queryfile> | read the query from an external file instead of passing it as an operand -h, --help | print help message -r, --raw | a complement to -c that returns raw (unquoted) strings when the result is an array of strings -u <url>, --url=<url> | add the base URL against which the HTML should be evaluated; influences the value of the DOM properties @.href, @.src when the HTML attributes are relative -V, --version | display the current version -x, --xml | parse the input as XML rather than HTML

A real-life example

Let's take a web page that uses atomic, presentational CSS rather than semantic CSS classes (and thus makes it more challenging to extract data), such as my starred repos page. To extract info about the repositories, at the time of writing:

curl https://github.com/danburzo\?tab\=stars | hred "
.mb-1 {
	h3 a ...{ 
		@href => url , 
		@.textContent => title 
	}, 
	^ :scope ~ .py-1 @.textContent => description 
}"

Let's break the query apart:

For each element with the class mb-1:

  1. on the one hand, find <a> elements nested into <h3>s:
    1. read their href HTML attribute as url and their textContent DOM property as title;
    2. merge the resulting object into the current scope with >> .;
  2. on the other hand, find the first (^) subsequent element (:scope ~) that matches the class py-1
    1. extract its textContent as description.

The resulting JSON, abridged:

[
  {
    "url": "/urfave/cli",
    "title": "\n        urfave / cli\n      ",
    "description": "\n      \n        A simple, fast, and fun package for building command line apps in Go\n      \n  "
  },

A note on security

hred uses as its DOM environment jsdom, which has the ability to run the JavaScript included in web pages. Because scripts specially crafted to attack jsdom may potentially evade the sandbox to which their execution is confined and access your machine through Node.js APIs, script execution is disabled; furthermore, external resources (scripts, images, stylesheets, iframes) are not fetched. Even with these precautions, be careful with what web pages you process with hred; when in doubt, inspect the page's source code beforehand.

Related projects

You might be interested in these:

  • pup was the original jq for HTML;
  • x-ray has the concept of including HTML attributes in the query string;
  • gdomqsx looks a bit like GraphQL, so maybe GraphQL for DOM can be a thing;
  • tq — another popular CLI tool for extracting data from HTML;
  • htmlq — like jq, but for HTML;
  • xidel supports a variety of query languages (CSS, XQuery, XPath, etc.);
  • wikipedia_ql — a query language for efficient data extraction from Wikipedia;
  • dbohdan/structured-text-tools maintains a comprehensive list of command-line tools for manipulating structured text data.