⚠️ WARNING: This project is currently under development.

🔌 HTTP(s) Agent

A light-weight, performant, highly composable and configurable blueprint for designing consistent and re-usable Node.js HTTP clients.

Legend

• Introduction
• Why use agent
• Install
• Usage
  • Transforming responses
• Performance
• Core design principles
• Why TypeScript?
  • Technical excellence and agile ways of working
  • Encourage Best Practices
• Node version support
  • Why ES2018?
• Testing
• TODO

Why use agent

... as opposed to request or node-fetch?

• request is/was great, but it has entered maintenance mode.
• Both node-fetch and request are relatively low-level (in JavaScript terms) implementations and as such lack certain convenience methods/APIs that help design maintainable and consistent HTTP clients. This is especially true in the microservices architecture context, where consistency is paramount.

agent builds on node-fetch to enable composable and re-usable HTTP(s) client implementations.

• Enforces a consistent approach to designing HTTP(s) clients.
• Greatly reduces common boilerplate such as
  • authentication,
  • default headers,
  • default options,
  • composing urls,
  • connection pooling,
  • parsing responses, and more.
• It is written in TypeScript.

Install

⚠️ NOTE: The TypeScript compiler is configured to target ES2018 and the library uses commonjs for module resolution (for now). Read more about Node version support.

npm install @asri/agent
# Additionally, for TypeScript users
npm install @types/node-fetch --save-dev

Usage

⚠️ WARNING: Unlike request, node-fetch does NOT reject non-ok responses by default as per the whatwg spec. If you wish, you can mimic this behaviour with a custom responseTransformer (see Transforming responses).

import { HttpClient, RequestInterceptor, ResponseTransformer } from "@asri/agent";

class GitHubClient extends HttpClient {
  constructor() {
    super({
      baseUrl: "https://api.github.com/",
      baseHeaders: { "accept": "application/vnd.github.v3+json" },
      // Set any `node-fetch` supported `Request` options.
      // Note that headers MUST be set in `baseHeaders`.
      baseOptions: { timeout: 5000 },
      // Automatically includes `accept: application/json` and 
      // `content-type: application/json` headers and parses responses to json.
      // It will also use the default `jsonResponseTransformer`
      // to parse responses into json. Does NOT reject non-ok responses.
      json: true,
    });
  }

  private static correlationIdHeader = "x-correlation-id";

  // Inspired by Apollo's REST Data Source, this lifecycle method
  // can be used to perform useful actions before a request is sent.
  protected willSendRequest: RequestInterceptor = (url, { headers }) => {
    const { correlationIdHeader } = HttpStatClient;

    if (!(correlationIdHeader in headers)) {
      console.warn(`missing ${correlationIdHeader} header`);
    };
  }

  // Finally, expose pre-configured, provider-specific methods to your consumers / API users.
  getOrganisationDetails = (orgId: string) => this.get(`/orgs/${orgId}`);

  getOrganisationRepositories = (orgId: string) => this.get(`/orgs/${orgId}/repos`, { timeout: 2500 });
}

// Initiate the client.
const gitHubClient = new GitHubClient();

// Resulting headers:
// `accept: application/vnd.github.v3+json` -> provided in `baseHeaders`
// `content-type: application/json` -> set internally due to `json: true`.
// Will warn because there is no `x-correlation-id` header set.
// Will use `timeout: 5000` as defined in `baseOptions`.
const organisationDetails = await gitHubClient.getOrganisationDetails();

// Same as above, but uses `timeout: 2500` – see `getOrganisationRepositories` implementation.
const organisationRepositories = await gitHubClient.getOrganisationRepositories();

Transforming responses

There is a great deal of discussion on whether fetch shuld or should not reject non-ok responses.

• https://github.com/whatwg/fetch/issues/18
• https://github.com/github/fetch/issues/155

We believe such design choices should ultimately be made by individual engineering teams, so the HttpClient base class exposes a convenient mechanism to transform responses via the transformResponse method.

import { HttpClient, RequestInterceptor, ResponseTransformer } from "@asri/agent";

class GitHubClient extends HttpClient {
  constructor() {
    super({ baseUrl: "https://api.github.com/", json: true });
  }

  private static correlationIdHeader = "x-correlation-id";

  protected willSendRequest: RequestInterceptor = (url, { headers }) => {
    const { correlationIdHeader } = HttpStatClient;

    if (!(correlationIdHeader in headers)) {
      console.warn(`missing ${correlationIdHeader} header`);
    }
  };

  // Mmimics the default behaviour of request, e.g. non-ok responses
  // are rejected rather than resolved.
  protected transformResponse: ResponseTransformer = async (response) => {
    // You need to be careful with 204 No Content, please consider
    // using our pre-built `jsonResponseTransformer` here instead.
    const jsonResponse = await response.json();

    if (response.ok) {
      return jsonResponse;
    } else {
      throw jsonResponse;
    }
  };

  getOrganisationDetails = (orgId: string) => this.get(`/orgs/${orgId}`);
}

const gitHubClient = new GitHubClient();

// Non-ok responses, for example 404, will now reject.
const organisationDetails = await gitHubClient.getOrganisationDetails();

Performance

Take advantage of a dramatic increase in throughput with agent – out-of-the-box!

With default request:

😰 Requests/sec: 362.19

wrk -t5 -c500 -d30s http://localhost:3001/request

Running 30s test @ http://localhost:3001/request
  5 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   660.35ms  389.35ms   1.50s    59.19%
    Req/Sec    87.36     66.56   350.00     80.29%
  10893 requests in 30.08s, 186.18MB read
  Socket errors: connect 254, read 109, write 0, timeout 0
Requests/sec:    362.19
Transfer/sec:      6.19MB

With default node-fetch:

😥 Requests/sec: 286.98

wrk -t5 -c500 -d30s http://localhost:3001/fetch

Running 30s test @ http://localhost:3001/fetch
  5 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   691.28ms  281.70ms   1.50s    76.50%
    Req/Sec    76.03     45.21   252.00     71.53%
  8632 requests in 30.08s, 147.54MB read
  Socket errors: connect 254, read 316, write 0, timeout 61
Requests/sec:    286.98
Transfer/sec:      4.90MB

With default agent (uses node-fetch):

🎉 Requests/sec: 2370.72

wrk -t5 -c500 -d30s http://localhost:3001/http-client

Running 30s test @ http://localhost:3001/
  5 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    70.92ms   57.73ms   1.90s    94.05%
    Req/Sec   555.13    164.19   770.00     84.07%
  71359 requests in 30.10s, 1.19GB read
  Socket errors: connect 254, read 377, write 0, timeout 77
Requests/sec:   2370.72
Transfer/sec:     40.52MB

These tests were all performed on an identical 2.4 GHz 8-Core Intel Core i9; 64 GB 2667 MHz DDR4 machine. We tested the different implementations of an inter-service HTTP GET request with request, node-fetch, and agent respectively. The services involved were running locally, and the payload consisted of a list of 100 users, each with 6 basic properties. The benchmarks ran for 30 seconds, using 5 threads and keeping 500 HTTP connections open.

Core design principles

• Code quality; The modules contained within this package may be used in mission-critical software, so it's important that the code is performant, secure, and battle-tested.

• Developer experience; Developers must be able to use this package with no significant barriers to entry. It has to be easy-to-find, well-documented, and pleasant to use.

• Modularity & Configurability; It's important that our colleagues can compose, and easily change the ways in which they use this package.

Encourage Best Practices

This collection of modules may serve as a reference implementation for many of our colleagues. It should therefore encourage best practices such as writing easy to understand code, comprehensive testing, automation, documentation, semantic versioning (+ semantic commit messages, automatic changelogs, etc.), up-to-date dependencies, and more.

TypeScript

In line with our guiding principles, this package is written in TypeScript.

While the use of TypeScript is not prescribed, it is worth noting that adopting it may result in increased productivity.

Technical excellence and agile ways of working

• Great tooling and overall developer experience. Strong and thriving open-source community, backed by Microsoft.

• Increased productivity. Type inference, intelligent code completion, and refactoring in confidence all contribute to increased productivity through minimising a specific class of bugs, reducing boilerplate, and maintaining a healthy codebase.

• A taste of future JavaScript, with optional types. Always up-to-date with upcoming ECMA features, compliant with proposals/specs.

• Helps attract and retain the best talent. TypeScript consistently ranks as one of the most loved and wanted languages in the annual StackOverflow developer surveys.

You can read more about TypeScript in the handbook.

Node version support

The TypeScript compiler is configured to target ES2018. In practice, this means projects consuming this package should run on Node 12 or higher, unless additional compilation/transpilation steps are in place to ensure compatibility with the target runtime.

Please see https://node.green/#ES2018 for reference.

Why ES2018?

Firstly, according to the official Node release schedule, Node 12.x entered LTS on 2019-10-21 and is scheduled to enter Maintenance on 2020-10-20. With the End-of-Life scheduled for April 2022, we are confident that most users will be running 12.x or higher.

Secondly, the 7.3 release of V8 (ships with Node 12.x or higher) includes "zero-cost async stack traces".

From the release notes:

We are turning on the --async-stack-traces flag by default. Zero-cost async stack traces make it easier to diagnose problems in production with heavily asynchronous code, as the error.stack property that is usually sent to log files/services now provides more insight into what caused the problem.

Testing

Ava and Jest were considered. Jest was chosen as it is easy to configure and includes most advanced features out of the box.

Prefer using Nock over mocking.

TODO

A quick and dirty tech debt tracker before we move to Issues.

• [ ] Write contributing guide
• [ ] Complete testing section, add best practices
• [ ] Describe scripts and usage, add best practices
• [ ] Add typespec and generate docs
• [ ] Describe security best practices, e.g. npm doctor, npm audit, npm outdated, ignore-scripts in .npmrc, etc.
• [ ] Add "Why should I use this" section
• [ ] Implement and document support for basic auth
• [ ] Document reponseTransformer
• [ ] Library architectural design (+ diagram?)