fetch-hooks
v4.1.2
Published
WhatWG-compatible `fetch` with extras, e.g. data:// and s3://
Downloads
25
Readme
fetch-hooks
Hook a WhatWG-compatible fetch
function with customised behaviour, e.g.:
More experimentally:
- Dumping
curl
command lines - Logging to
rsyslog
- Adding your own custom lifecycle hooks
Being considered:
Usage
const { hook, fetch, hooks } = require('fetch-hooks');
const _fetch = hook(fetch, myHook); // construct a hooked fetch
const response = await _fetch(uri); // and use it as if it were normal
console.log(await response.text());
Hooking data:
URLs for methods GET
and HEAD
Add hooks.data
to support data:
URIs:
const _fetch = hook(fetch, hooks.data);
const response = await _fetch('data:text/ascii;base64,TUlORCBCTE9XTg==');
console.log(await response.text());
Hooking file:
URLs for methods GET
and HEAD
Add a hooks.file
return value to support S3 bucket access via s3:
URIs:
const _fetch = hook(fetch, hooks.file({ baseURI: process.cwd() }));
const response = await _fetch('file:test/data/smiley.txt');
console.log(await response.text());
File hooks are only active for request URIs within baseURI
, which defaults to the process' current working directory at request time.
WARNINGS:
Due to quirks of node-fetch
2.1.1 (2018-03-05):
If you request a relative
file:
URI,hook
will resolve it against the process' current working directory at request time before passing it to the hooks. This is necessary to survive theRequest
constructor.If you call
response.text()
, all files are read as if encoded in UTF-8, even if they aren't. I'm jammingcharset=UTF-8
into theContent-Type
as fair warning. Try thenode-fetch
extensionsresponse.body.buffer
andresponse.body.textConverted()
if this doesn't work for you.
Hooking s3:
URLs for methods GET
, HEAD
, PUT
, and DELETE
Add a hooks.s3
return value to support S3 bucket access:
// construct an S3 Service object
const { S3 } = require('aws-sdk');
const s3 = new S3({
region: 'ap-southeast-2',
signatureVersion: 'v4',
});
// attach it to an S3 hook for your bucket's base URL:
const { hook, fetch, hooks } = require('fetch-hooks');
const s3hook = hooks.s3(s3, { baseURI: 's3://bucket', acl: 'private' });
const _fetch = hook(fetch, s3hook);
// fetch content from the bucket
const response = await _fetch('s3://bucket/key');
console.log(await response.text());
S3 URIs give the bucket name where you'd expect the host name, consistent with the aws s3
command line.
S3 hooks are only active for request URIs within baseURI
, which defaults to s3:/
to match any bucket.
The acl
option specifies a canned ACL, and defaults to private
.
Enforcing HTTPS
Add hooks.httpsOnly
to enforce that you're only willing to speak over the HTTPS protocol.
const { hook, fetch, hooks } = require('fetch-hooks');
const _fetch = hook(fetch, /* other hooks, */ hooks.httpsOnly);
It's safe to use hooks.httpsOnly
last in the chain, as the file:
and data:
hooks will have already responded, and the s3:
hook will have changed the request to a signed https:
request.
Troubleshooting with curl
Add hooks.curl
to write curl
commands to standard error:
const _fetch = hook(fetch, /* other hooks, */ hooks.curl);
You can also set the DEBUG
environment variable to have useful information dumped to the console. See the debug
documentation for more detail.
WARNINGS:
This part of the API is not yet stable, the input handling in particular. I reserve the right to make breaking changes with minor or patch level version bumps until I see some sign of third party usage. I welcome PRs and suggestions.
The commands assume any input for
PUT
,POST
etc are in/tmp/input
. The hook does not write any such content to/tmp/input
.
Logging to a remote syslog server
Add hooks.rsyslog
to send packets to an RFC5424 compliant server.
const _fetch = hook(fetch, /* other hooks, */ hooks.rsyslog({
target_host: '127.0.0.1',
target_port: 514,
elide: url => withPartsRemoved(url),
}));
elide
is optional. The default for elide
will remove, from the URLs
sent to the remote syslog:
- The
auth
component - The
query
component - The data in the
pathname
component, if the protocol isdata:
Otherwise put, the default elide
preserves only:
protocol
host
, which includes the port numberpathname
, unlessprotocol
isdata:
I chose a default this conservative so neither of us have to scramble to remove usernames, passwords, and secrets embedded in queries from our log files.
To make your own choices, override elide
with a function returning a string given a URL. The following will pass the full URL:
const _fetch = hook(fetch, /* other hooks, */ hooks.rsyslog({
target_host: '127.0.0.1',
target_port: 514,
elide: url => url,
}));
For full documentation of the rest of the options, see the rsyslog
package.
WARNINGS:
This part of the API is not yet stable, the output format in particular. I reserve the right to make breaking changes with minor or patch level version bumps until I see some sign of third party usage. I welcome PRs and suggestions.
The
len=
segment requires some guesswork, and might not match the number of bytes on the wire, especially if the server omits or lies about thecontent-length
.
Adding your own lifecycle hooks
Return a function named prereq
, postreq
, or error
to get called either before a request, after a request, or when a hook fails:
postreq(req, res, err)
will be called after a request is made, with eitherres
orerr
set tonull
depending on whether the request crashed out or succeeded.error(err)
will be called if a call to a hook or lifecycle hook function fails.
WARNINGS:
- This part of the API is not yet stable. I reserve the right to make breaking changes with minor or patch level version bumps until I see some sign of third party usage. I welcome PRs and suggestions.
Under the Covers
const { hook, fetch, hooks } = require('fetch-hooks');
const _fetch = hook(fetch /* , ... hooks */);
const response = await _fetch(uri);
console.log(await response.text());
A hooked fetch
will construct a Request
using node-fetch
and then, for each of its hooks:
- Call the hook
await
its return value- Ignore the hook if the return value is falsey
- Resolve with the return value's
response
property if found - Continue with the return value's
request
property if found
If there are no more hooks, a hooked fetch
will:
- Pass through to its upstream
fetch
(its first argument) if no hooks are left, or - Reject with an error if its upstream
fetch
isnull
Typings
I've cloned the Microsoft typings for fetch
and related types, adapting them for node-fetch
. If the typings don't match the node-fetch
reality, please open an issue.
Previous experiments that didn't work:
Relying on the Microsoft typings directly, i.e. requiring
dom
incompilerOptions.lib
intsconfig.json
. The global namespace clutter fromdom
made it hard to find undefined variables,fetch
in particular.Relying on
@types/node-fetch
. I didn't enjoy the mismatch with Microsoft's typings forfetch
, and didn't have the time to encourage the breaking changes required to track closer. They're looking better, now, so it might be worth revisiting this in the future.
Background
A few small things bother me about using WhatWG fetch
for back end programming:
I can't use
fetch
for protocols other thanhttp:
andhttps:
, making it hard to use when I'm receiving a trusted URI that could reasonably have afile:
ordata:
protocol.If my packages take a
fetch
as part of their configuration, I need to mockfetch
during tests. I'd like that to be easier.If they don't, I have to have them take an
agent
as part of their configuration if they expect HTTPS. I also have to stand up an HTTPS server.It'd sometimes be handy to apply different outbound headers for different hosts, but I don't want to make the code making the requests responsible for that application.
This repository is an experiment which might end up in proof by contradiction. My premise is:
- An API-compatible
fetch
with the ability to register hooks could solve all the above, and possibly open some exciting possibilities.