npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

fetch-hooks

v4.1.2

Published

WhatWG-compatible `fetch` with extras, e.g. data:// and s3://

Downloads

25

Readme

fetch-hooks

Build status Greenkeeper badge

Hook a WhatWG-compatible fetch function with customised behaviour, e.g.:

  • Handling data: URIs
  • Handling file: URLs
  • Handling s3: URLs
  • Enforcing https: as your transport protocol

More experimentally:

Being considered:

Usage

const { hook, fetch, hooks } = require('fetch-hooks');
const _fetch = hook(fetch, myHook); // construct a hooked fetch
const response = await _fetch(uri); // and use it as if it were normal
console.log(await response.text());

Hooking data: URLs for methods GET and HEAD

Add hooks.data to support data: URIs:

const _fetch = hook(fetch, hooks.data);
const response = await _fetch('data:text/ascii;base64,TUlORCBCTE9XTg==');
console.log(await response.text());

Hooking file: URLs for methods GET and HEAD

Add a hooks.file return value to support S3 bucket access via s3: URIs:

const _fetch = hook(fetch, hooks.file({ baseURI: process.cwd() }));
const response = await _fetch('file:test/data/smiley.txt');
console.log(await response.text());

File hooks are only active for request URIs within baseURI, which defaults to the process' current working directory at request time.

WARNINGS:

Due to quirks of node-fetch 2.1.1 (2018-03-05):

  • If you request a relative file: URI, hook will resolve it against the process' current working directory at request time before passing it to the hooks. This is necessary to survive the Request constructor.

  • If you call response.text(), all files are read as if encoded in UTF-8, even if they aren't. I'm jamming charset=UTF-8 into the Content-Type as fair warning. Try the node-fetch extensions response.body.buffer and response.body.textConverted() if this doesn't work for you.

Hooking s3: URLs for methods GET, HEAD, PUT, and DELETE

Add a hooks.s3 return value to support S3 bucket access:

// construct an S3 Service object
const { S3 } = require('aws-sdk');
const s3 = new S3({
    region: 'ap-southeast-2',
    signatureVersion: 'v4',
});

// attach it to an S3 hook for your bucket's base URL:
const { hook, fetch, hooks } = require('fetch-hooks');
const s3hook = hooks.s3(s3, { baseURI: 's3://bucket', acl: 'private' });
const _fetch = hook(fetch, s3hook);

// fetch content from the bucket
const response = await _fetch('s3://bucket/key');
console.log(await response.text());

S3 URIs give the bucket name where you'd expect the host name, consistent with the aws s3 command line.

S3 hooks are only active for request URIs within baseURI, which defaults to s3:/ to match any bucket.

The acl option specifies a canned ACL, and defaults to private.

Enforcing HTTPS

Add hooks.httpsOnly to enforce that you're only willing to speak over the HTTPS protocol.

const { hook, fetch, hooks } = require('fetch-hooks');
const _fetch = hook(fetch, /* other hooks, */ hooks.httpsOnly);

It's safe to use hooks.httpsOnly last in the chain, as the file: and data: hooks will have already responded, and the s3: hook will have changed the request to a signed https: request.

Troubleshooting with curl

Add hooks.curl to write curl commands to standard error:

const _fetch = hook(fetch, /* other hooks, */ hooks.curl);

You can also set the DEBUG environment variable to have useful information dumped to the console. See the debug documentation for more detail.

WARNINGS:

  • This part of the API is not yet stable, the input handling in particular. I reserve the right to make breaking changes with minor or patch level version bumps until I see some sign of third party usage. I welcome PRs and suggestions.

  • The commands assume any input for PUT, POST etc are in /tmp/input. The hook does not write any such content to /tmp/input.

Logging to a remote syslog server

Add hooks.rsyslog to send packets to an RFC5424 compliant server.

const _fetch = hook(fetch, /* other hooks, */ hooks.rsyslog({
  target_host: '127.0.0.1',
  target_port: 514,
  elide: url => withPartsRemoved(url),
}));

elide is optional. The default for elide will remove, from the URLs sent to the remote syslog:

  • The auth component
  • The query component
  • The data in the pathname component, if the protocol is data:

Otherwise put, the default elide preserves only:

  • protocol
  • host, which includes the port number
  • pathname, unless protocol is data:

I chose a default this conservative so neither of us have to scramble to remove usernames, passwords, and secrets embedded in queries from our log files.

To make your own choices, override elide with a function returning a string given a URL. The following will pass the full URL:

const _fetch = hook(fetch, /* other hooks, */ hooks.rsyslog({
  target_host: '127.0.0.1',
  target_port: 514,
  elide: url => url,
}));

For full documentation of the rest of the options, see the rsyslog package.

WARNINGS:

  • This part of the API is not yet stable, the output format in particular. I reserve the right to make breaking changes with minor or patch level version bumps until I see some sign of third party usage. I welcome PRs and suggestions.

  • The len= segment requires some guesswork, and might not match the number of bytes on the wire, especially if the server omits or lies about the content-length.

Adding your own lifecycle hooks

Return a function named prereq, postreq, or error to get called either before a request, after a request, or when a hook fails:

  • postreq(req, res, err) will be called after a request is made, with either res or err set to null depending on whether the request crashed out or succeeded.

  • error(err) will be called if a call to a hook or lifecycle hook function fails.

WARNINGS:

  • This part of the API is not yet stable. I reserve the right to make breaking changes with minor or patch level version bumps until I see some sign of third party usage. I welcome PRs and suggestions.

Under the Covers

const { hook, fetch, hooks } = require('fetch-hooks');
const _fetch = hook(fetch /* , ... hooks */);
const response = await _fetch(uri);
console.log(await response.text());

A hooked fetch will construct a Request using node-fetch and then, for each of its hooks:

  • Call the hook
  • await its return value
  • Ignore the hook if the return value is falsey
  • Resolve with the return value's response property if found
  • Continue with the return value's request property if found

If there are no more hooks, a hooked fetch will:

  • Pass through to its upstream fetch (its first argument) if no hooks are left, or
  • Reject with an error if its upstream fetch is null

Typings

I've cloned the Microsoft typings for fetch and related types, adapting them for node-fetch. If the typings don't match the node-fetch reality, please open an issue.

Previous experiments that didn't work:

  • Relying on the Microsoft typings directly, i.e. requiring dom in compilerOptions.lib in tsconfig.json. The global namespace clutter from dom made it hard to find undefined variables, fetch in particular.

  • Relying on @types/node-fetch. I didn't enjoy the mismatch with Microsoft's typings for fetch, and didn't have the time to encourage the breaking changes required to track closer. They're looking better, now, so it might be worth revisiting this in the future.

Background

A few small things bother me about using WhatWG fetch for back end programming:

  • I can't use fetch for protocols other than http: and https:, making it hard to use when I'm receiving a trusted URI that could reasonably have a file: or data: protocol.

  • If my packages take a fetch as part of their configuration, I need to mock fetch during tests. I'd like that to be easier.

  • If they don't, I have to have them take an agent as part of their configuration if they expect HTTPS. I also have to stand up an HTTPS server.

  • It'd sometimes be handy to apply different outbound headers for different hosts, but I don't want to make the code making the requests responsible for that application.

This repository is an experiment which might end up in proof by contradiction. My premise is:

  • An API-compatible fetch with the ability to register hooks could solve all the above, and possibly open some exciting possibilities.