redis-streams-cleaner

Clean Redis Streams messages that are were already read by all consumer groups

Project purpose

Redis Streams are one of the great features of Redis Server.
BUT currently there is no easy way to clear messages on a stream that were already read by all consumer groups.

This library aims to give you an easy way to keep your Redis Streams as little as possible, by creating a safe way to clear messages that were already read by everybody that should do it.

Keep in mind that this library assumes that you use consumer groups (single or multiple) to read messages from Redis Streams. It is also assumed that messages ids are autogenerated by Redis Stream (timestamps with sequences).

Usage

Install it using a node package manager

npm install --save redis-streams-cleaner

or

yarn add redis-streams-cleaner

You can create a small program like below and run it on schedules (for instance in a cron).
The option timeToKeepBeforeLastProcessedMessage (default: 0) tells the clean up to keep some time range (in milliseconds) of messages after the last message that was already read by all its consumers.

import { RedisStreamCleaner } from "redis-streams-cleaner";

import * as Redis from "redis";

const redisClient = Redis.createClient({
  host: "localhost",
  port: 6379,
  db: 1,
});

setImmediate(async () => {
  const redisStreamCleaner = new RedisStreamCleaner(redisClient, {
    timeToKeepBeforeLastProcessedMessage: 0, //time in ms
  });

  console.log("Starting Redis Streams Clean up!");
  await redisStreamCleaner.cleanStreamMessages("TEST_STREAM");
  console.log("Done!");

  process.exit(0);
});

Tech stack

• Typescript
• Yarn
• Makefile
• Redis
• Redis Streams
• Jest
• Docker
• Docker Compose

Installing build tools

make tools

Installing dependencies

make install

How to build

make build

How to run unit tests

make test

How to run functional tests

Functional tests use Docker and Docker Compose to create a testing sandbox (a Redis Server).

make functional-test