npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@hqoss/agent

v0.0.0-alpha.10

Published

A light-weight, performant, composable blueprint for writing consistent and re-usable Node.js HTTP clients.

Downloads

2

Readme

Node.js CI Codacy Badge Codacy Badge

🔌 Node Agent

A light-weight, performant, composable blueprint for writing consistent and re-usable Node.js HTTP clients.

Extends node-fetch, therefore 100% compatible with the underlying APIs.

Table of contents

🤔 Why use agent

... as opposed to request or node-fetch?

  • request is/was great, but it has entered maintenance mode.
  • Both node-fetch and request are relatively low-level (in JavaScript terms) implementations and as such lack certain convenience methods/APIs that help design maintainable and consistent HTTP clients. This is especially true in the microservices architecture context, where consistency is paramount.

agent builds on node-fetch to enable composable and re-usable HTTP client implementations.

  • Enforces a consistent approach to writing HTTP clients.

  • Greatly reduces common boilerplate, expressly

    • authentication,
    • default headers,
    • default options,
    • composing urls,
    • connection pooling,
    • parsing responses, and more.
  • It is written in TypeScript.

⏳ Install

⚠️ NOTE: The project is configured to target ES2018 and the library uses commonjs module resolution. Read more in the Node version support section.

npm install @hqoss/agent
# Additionally, for TypeScript users
npm install @types/node-fetch --save-dev

📝 Usage

⚠️ WARNING: Unlike request, agent (using node-fetch under the hood) does NOT reject non-ok responses by default as per the whatwg spec. You can, however, mimic this behaviour with a custom responseTransformer (see Transforming responses).

Basic

Define:

import { HttpClient } from "@asri/agent";

class GitHubClient extends HttpClient {
  constructor() {
    super({
      baseUrl: "https://api.github.com/",
      baseHeaders: { "accept": "application/vnd.github.v3+json" },
      // Set any `node-fetch` supported `Request` options.
      // Note that headers MUST be set in `baseHeaders`.
      baseOptions: { timeout: 5000 },
      // Automatically includes `accept: application/json` and 
      // `content-type: application/json` headers and parses responses to json.
      // It will also use the default `jsonResponseTransformer`
      // to parse responses into json. Does NOT reject non-ok responses.
      json: true,
    });
  }

  // Expose pre-configured, provider-specific methods to your consumers / API users.
  getOrganisationDetails = (orgId: string) => this.get(`/orgs/${orgId}`);

  getOrganisationRepositories = (orgId: string) => this.get(`/orgs/${orgId}/repos`, { timeout: 2500 });
}

export default GitHubClient;

Consume:

// Suppose you have a shared library for Http clients.
import { GitHubClient } from "@clients/github";

// You can also use the client's constructor to provide additional configuration here.
const gitHubClient = new GitHubClient();

// Resulting headers:
// `accept: application/vnd.github.v3+json` -> provided in `baseHeaders`
// `content-type: application/json` -> set internally due to `json: true`.
// Will warn because there is no `x-correlation-id` header set.
// Will use `timeout: 5000` as defined in `baseOptions`.
const organisationDetails = await gitHubClient.getOrganisationDetails();

// Same as above, but uses `timeout: 2500` – see `getOrganisationRepositories` implementation.
const organisationRepositories = await gitHubClient.getOrganisationRepositories();

Intercepting requests

You can intercept every request by implementing the willSendRequest lifecycle method.

import { HeaderKey, HttpClient, RequestInterceptor } from "@asri/agent";

class GitHubClient extends HttpClient {
  constructor() {
    super({
      baseUrl: "https://api.github.com/",
      baseHeaders: { "accept": "application/vnd.github.v3+json" },
      json: true,
    });
  }

  private static correlationIdHeader = HeaderKey.CorrelationId;

  // Inspired by Apollo's REST Data Source, this lifecycle method
  // can be used to perform useful actions before a request is sent.
  protected willSendRequest: RequestInterceptor = (url, { headers }) => {
    const { correlationIdHeader } = HttpStatClient;

    console.info(`Outgoing request to ${url}`);

    if (!(correlationIdHeader in headers)) {
      console.warn(`missing ${correlationIdHeader} header`);
    };
  }

  // ... pre-configured methods follow.
}

export default GitHubClient;

Transforming responses

There is a great deal of discussion on whether fetch should or should not reject non-ok responses [1,2]. We believe such design choices should ultimately be made by the consumers, so the HttpClient base class exposes a convenient mechanism to transform responses via the transformResponse method.

import { HttpClient, ResponseTransformer } from "@asri/agent";

class GitHubClient extends HttpClient {
  constructor() {
    super({ baseUrl: "https://api.github.com/", json: true });
  }

  // Mmimics the default behaviour of request, e.g. non-ok responses
  // are rejected rather than resolved.
  protected transformResponse: ResponseTransformer = async (response) => {
    // You need to be careful with 204 No Content, please consider
    // using our pre-built `jsonResponseTransformer` here instead.
    const jsonResponse = await response.json();

    if (response.ok) {
      return jsonResponse;
    } else {
      throw jsonResponse;
    }
  };

  // ... pre-configured methods follow.
}

export default GitHubClient;

Consume:

// Suppose you have a shared library for Http clients.
import { GitHubClient } from "@clients/github";

// You can also use the client's constructor to provide additional configuration here.
const gitHubClient = new GitHubClient();

// Non-ok responses, for example 404, will now reject.
const organisationDetails = gitHubClient.getOrganisationDetails()
  .then(console.log)
  // A non-ok response will now end up here.
  .catch(console.error);

⚡️ Performance

We ship the default HttpClient with a pre-configured (Node.js) Agent, which may lead to a huge increase in throughput.

For reference, we performed a number of benchmarks comparing the out-of-the-box request, node-fetch, and agent clients. To fetch a list of 100 users from one service to another (see diagram below), these were the results:

| wrk | -HTTP-> | Server A -> HttpClient | -HTTP-> | Server B -> data in memory |
  • Default request setup (used by most projects): 10,893 requests in 30.08s; 362.19 requests/sec
  • Default node-fetch setup (used by many projects): 8,632 requests in 30.08s; 286.98 requests/sec
  • Default agent setup: 71,359 requests in 30.10s; 2,370.72 requests/sec

Please note that these benchmarks were run through wrk, each lasting 30 seconds, using 5 threads and keeping 500 connections open.

This is the default Agent configuration, which can easily be overriden in the HttpClient constructor. You can simply provide your own Agent instance in baseOptions.

const opts = {
  keepAlive: true,
  maxSockets: 64,
  keepAliveMsecs: 5000,
};

🧬 Core design principles

  • Code quality; This package may end up being used in mission-critical software, so it's important that the code is performant, secure, and battle-tested.

  • Developer experience; Developers must be able to use this package with no significant barriers to entry. It has to be easy-to-find, well-documented, and pleasant to use.

  • Modularity & Configurability; It's important that users can compose and easily change the ways in which they consume and work with this package.

Node version support

The project is configured to target ES2018. In practice, this means consumers should run on Node 12 or higher, unless additional compilation/transpilation steps are in place to ensure compatibility with the target runtime.

Please see https://node.green/#ES2018 for reference.

Why ES2018

Firstly, according to the official Node release schedule, Node 12.x entered LTS on 2019-10-21 and is scheduled to enter Maintenance on 2020-10-20. With the End-of-Life scheduled for April 2022, we are confident that most users will now be running 12.x or higher.

Secondly, the 7.3 release of V8 (ships with Node 12.x or higher) includes "zero-cost async stack traces".

From the release notes:

We are turning on the --async-stack-traces flag by default. Zero-cost async stack traces make it easier to diagnose problems in production with heavily asynchronous code, as the error.stack property that is usually sent to log files/services now provides more insight into what caused the problem.

❤️ Testing

Ava and Jest were considered. Jest was chosen as it is very easy to configure and includes most of the features we need out-of-the-box.

Further investigation will be launched in foreseeable future to consider moving to Ava.

We prefer using Nock over mocking.

TODO

A quick and dirty tech debt tracker before we move to Issues.

  • [ ] Write a Contributing guide
  • [ ] Complete testing section, add best practices
  • [ ] Describe scripts and usage, add best practices
  • [ ] Add typespec and generate docs
  • [ ] Describe security best practices, e.g. npm doctor, npm audit, npm outdated, ignore-scripts in .npmrc, etc.
  • [ ] Add "Why should I use this" section
  • [ ] Implement and document support for basic auth
  • [ ] Document willSendRequest and reponseTransformer
  • [ ] Library architectural design (+ diagram?)