npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

hfxbus

v2.2.0

Published

Redis backed high frequency exchange bus

Downloads

8

Readme

HFXBus

HFXBus is a bus implementation for NodeJS backed by Redis Streams and PubSub.

  • Focused on performance for asynchronous communication between endpoints
  • Replaces Kafka, RabbitMQ and other brokers
  • Support Redis Standalone or Cluster
  • Support for correct message acknowledgement, resolution and rejection states
  • Message's payload can be raw Buffer or JSON
  • Claiming logic to retry and collect stalled out messages
  • Architecture and payload agnostic
  • Limit the number of parallel processing done by your microservices
  • Supports client side partitioning to achieve Redis HA
  • Supports distributed routing to distribute streams across nodes
  • Unit and E2E tested

It's simple and effective to achieve high performance event-sourcing environment and microservice communication.

npm install --save hfxbus

Upgrading

This project was rewritten in Typescript on v2, if you're running v1 and need reference please visit the branch v1. The v2+ still defined as RC and will only achieve GA when we finish the tests for HFXBus v2.


How it works

HFXBus uses Redis Streams to enqueue messages and groups to consume messages, but these streams only controls the flow of messages to make the processing lighweight in networking and memory/CPU aspects. All payload is stored as regular Redis keys and it's up to your endpoints decide which keys need to be loaded/created.

Redis PubSub is used to emit events happening to messages like when a message is consumed, so your endpoints have feedback about messages events.

And finally, with XTRIM you can keep your Redis server memory utilization low and with XCLAIM improve your (micro)services redundancy/resilience. We implemented the command XRETRY using Lua Scripting to achieve a reliable way to retry stalled out messages.


Client side partitioning

HFXBus provides client side partitioning through the method ConnectionManager.nodes(), but you need to be aware of the following points:

  • This partitioning is efficient for partitioning job's payload.
  • Consumers group can't read from streams spread through multiple nodes, so you'll need to make them as static routing using the route parameter of consumers and the staticRoutes of connection parameters.
  • Always use the sequence parameter to specify the sequence of nodes when partitioning the data.
  • Producer's listen to Pub/Sub from a single connection only if the stream name is not a pattern, otherwise it'll listen to all nodes.

This feature was designed to work with the following architecture:

client side partitioning

Distributed routing

To really scale Redis horizontally using the architecture above HFXBus provides a routing method named "distributed routing":

  • Consumers will try to acquire messages from all connections using a round-robin algorithm to distribute the workload.
  • Producers will not more send messages using the specified route, instead they will use the job's ID as route.
  • You don't need to specify static routes.

But there are tradeoffs with this method:

  • Stream messages IDs can be repeated across nodes.
  • The number of connections done by each consumer increases.
  • Currently HFXBus doesn't auto eject failing connections (maybe you want to open a PR for this feature?).

Quick Start

First, setup a Redis running at 127.0.0.1:6379 (you can use docker). And then create a consumer.ts file with the following content:

import { ConnectionManager, Consumer } from 'hfxbus';

const connection = ConnectionManager.standalone({
  port: 6379,
  host: '127.0.0.1'
});

const consumer = new Consumer(connection, { group: 'worldConcat' });

consumer.process({
  stream: 'concat',
  processor: async (job) => {
    
    console.log(`Received job: ${job.id}`);

    const {
      inbound
    } = await job.get('inbound', false).del('inbound').pull();

    console.log(`Received inbound: ${inbound}`);

    await job.set('outbound', `${inbound} world!`).push();

    console.log('Job consumed');

  }
});

consumer.play().then(() => {
  console.log(`Consumer is waiting for jobs (consumer id is ${consumer.id})`);
}).catch((error) => console.error(error));

And another file as producer.ts:

import { ConnectionManager, Producer } from 'hfxbus';

const connection = ConnectionManager.standalone({
  port: 6379,
  host: '127.0.0.1'
});

const producer = new Producer(connection);

const execute = async () => {
  
  await producer.listen();

  console.log(`Producer is listening for messages (producer id is ${producer.id})`);

  const job = producer.job();

  console.log(`Created job: ${job.id}`);

  await job.set('inbound', 'Hello').push();

  await producer.send({
    stream: 'concat',
    waitFor: [
      'worldConcat'
    ],
    job
  });

  console.log(`Sent job: ${job.id}`);
  
  await job.finished();

  console.log(`Finished job: ${job.id}`);

  const {
    outbound
  } = await job.get('outbound', false).del('outbound').pull();

  console.log(`Outbound is: ${outbound}`);

}

execute().catch((error) => console.error(error));

Remember to start consumer.ts before producer.ts as by default consumer will receive only new jobs, you can change this behavior, take a look at the API Documentation.


API Documentation

Your can learn more about HFXBus API clicking here.


Related Projects

  • HFXWorker - A worker pool using NodeJS worker threads.
  • HFXEventStash - A high performance event store to persist commands (CQRS).