Node Redis key scanner

A simple ES6 Redis key scanner for Node 10 and newer. This is a small class that allows you to do one thing quickly and easily: scan a Redis key space for keys that match a given pattern.

See the Redis SCAN command documentation for information about how to write patterns for matching, the guarantees, caveats, etc.

Install

With NPM:

npm install node-redis-scan

Or with Yarn:

yarn add node-redis-scan

Use

Instantiate this class with a Node Redis client (version 3.x) then perform key space scans!

⚠️ Note: Node Redis clients of version 4.x or newer are not supported by this library. At least not yet. Luckily, if you are using version 4.x you may not need this library. You could likely use a Node Redis 4.x scan iterator instead of this library.

Redis supports scanning the entire key space or scanning hashes, sets, and sorted sets with the HSCAN, SSCAN, and ZSCAN commands, respectively. All of this functionality is available by calling the appropriately named functions listed below.

Table of contents

• scan()
• eachScan()
• hscan(), eachHScan()
• sscan(), eachSScan()
• zscan(), eachZScan()
• Canceling a scan before it has finished

The scan() method

The scan() method provides the easiest way to scan your key space with a single callback that will be passed all matching keys. Depending on the size of your key space (millions of keys and beyond) this process might take many seconds or longer.

Parameters

|Name|Type|Description| |-|-|-| |pattern|string|The Redis glob-style string pattern to match keys against.| |options|object (optional)|An object for configuring the precise scan parameters. Available options:method - String name for which underlying Redis scan method we want to use. Defaults to 'scan' and can be set to one of 'hscan', 'sscan', or 'zscan'.key - The string name of the applicable key. Required if the method is set to 'hscan', 'sscan', or 'zscan'.count - A number representing how much work Redis should do with each iteration of the given scan command. This is useful if you want to scan a huge key space faster. The trade off is lengthening the brief segments of time that Redis is locked doing work scanning. See the Redis COUNT option documentation.type - A string name of a Redis key type. This is used for searching for keys of a certain type. See the Redis TYPE option documentation.limit - A number representing a limit for how many results should be returned. Because of the nature of the Redis SCAN command plus the interaction with the count option we can never guarantee returning this exact limit. When the limit is reached or exceeded the scan halts and is considered complete.| |callback|function|Invoked with (err, matchingKeys).|

Example

const redis = require('redis');
const redisScan = require('node-redis-scan');

const client = redis.createClient();
const scanner = new redisScan(client);

scanner.scan('some-pattern*', (err, matchingKeys) => {
    if (err) throw(err);

    // matchingKeys will be an array of strings if matches were found
    // otherwise it will be an empty array.
    console.log(matchingKeys);
});

// Or, with a custom COUNT option...
scanner.scan('*another-pattern', {count: 1000}, (err, matchingKeys) => {
    if (err) throw(err);

    console.log(matchingKeys);
});

The eachScan() method

The eachScan() method is useful if you want to perform work with matched keys at the same time as the key space is being scanned. When you’re scanning an enormous key space this is likely a more efficient way to operate: you can begin handling matched keys asynchronously, before the entire scan has finished. Unfortunately this approach doesn’t help in situations where you need to have every matching key prior to performing the next step in your operation/application.

Matching keys are passed to the intermediate callback function after each iteration of the Redis SCAN command. The final callback is passed a count of how many matching keys were returned.

Parameters

|Name|Type|Description| |-|-|-| |pattern|string|The Redis glob-style string pattern to match keys against.| |options|object (optional)|An object for configuring the precise scan parameters. Available options:method - String name for which underlying Redis scan method we want to use. Defaults to 'scan' and can be set to one of 'hscan', 'sscan', or 'zscan'.key - The string name of the applicable key. Required if the method is set to 'hscan', 'sscan', or 'zscan'.count - A number representing how much work Redis should do with each iteration of the given scan command. This is useful if you want to scan a huge key space faster. The trade off is lengthening the brief segments of time that Redis is locked doing work scanning. See the Redis COUNT option documentation.type - A string name of a Redis key type. This is used for searching for keys of a certain type. See the Redis TYPE option documentation.limit - A number representing a limit for how many results should be returned. Because of the nature of the Redis SCAN command plus the interaction with the count option we can never guarantee returning this exact limit. When the limit is reached or exceeded the scan halts and is considered complete.| |eachScanCallback|function|This intermediate callback is used for handling matching keys as they are returned. This function can also signal cancellation of the overall scan, if desired, by returning boolean true.Invoked with (matchingKeys).| |callback|function|Invoked with (err, matchCount).|

Example

const redis = require('redis');
const redisScan = require('node-redis-scan');

const client = redis.createClient();
const scanner = new redisScan(client);

scanner.eachScan('some-pattern*', (matchingKeys) => {
    // Depending on the pattern being scanned for, many or most calls to
    // this function will be passed an empty array.
    if (matchingKeys.length) {
        // Matching keys found after this iteration of the SCAN command.
        console.log(matchingKeys);
    }
}, (err, matchCount) => {
    if (err) throw(err);

    // matchCount will be an integer count of how many total keys
    // were found and passed to the intermediate callback.
    console.log(`Found ${matchCount} keys.`);
});

The hscan() and eachHScan() methods

Using hscan() will return all matching keys along with their values from the given hash. Note that the nature of HSCAN is to return the keys and their values.

Example

// Create a new instance, then:

scanner.hscan('name-of-hash', 'some-pattern*', (err, matchingKeysValues) => {
    if (err) return done(err);

    // matchingKeysValues will be an array of strings if matches were found
    // in the hash, otherwise it will be an empty array.
    console.log(matchingKeysValues);
});

// When working with an enormous hash you might prefer the
// `eachHScan()` approach, which is similar to `eachScan()`
// in that it lets you work with matches as they are returned.

scanner.eachHScan('name-of-hash', 'some-pattern*', (matchingKeysValues) => {
    // Depending on the pattern being scanned for, many or most calls to
    // this function will be passed an empty array.
    if (matchingKeysValues.length) {
        // Matching keys and values of the hash found after this
        // iteration of the HSCAN command.
        console.log(matchingKeysValues);
    }
}, (err, matchCount) => {
    if (err) throw(err);

    // matchCount will be an integer count of how many total keys
    // and values were found and passed to the intermediate callback.
    console.log(`Found ${matchCount} keys and values.`);
});

The sscan() and eachSScan() methods

Using sscan() will return all matching members from the given set.

Example

// Create a new instance, then:

scanner.sscan('name-of-set', 'some-pattern*', (err, matches) => {
    if (err) return done(err);

    // matches will be an array of strings if matches were found
    // in the set, otherwise it will be an empty array.
    console.log(matches);
});

// When working with an enormous set you might prefer the
// `eachSScan()` approach, which is similar to `eachScan()`

The zscan() and eachZScan() methods

Using zscan() will return all matching members along with their scores from a given sorted set. Note that the nature of ZSCAN is to return the members and their scores.

Example

// Create a new instance, then:

scanner.zscan('name-of-sorted-set', 'some-pattern*', (err, matchingMembersScores) => {
    if (err) return done(err);

    // matchingMembersScores will be an array of strings if matches were found
    // in the sorted set, otherwise it will be an empty array.
    console.log(matchingMembersScores);
});

// When working with an enormous sorted set you might prefer the
// `eachZScan()` approach, which is similar to `eachScan()`

Canceling a scan before it has finished

Using the limit option with either the scan() or eachScan() method allows us to halt or cancel a scan after a certain "limit" of matching keys have been found. Note that we might receive more matching keys than the specified limit because of the nature of the Redis SCAN command.

Example

scanner.scan('some-pattern*', {limit: 5}, (err, matchingKeys) => {
    if (err) throw(err);

    // We'll have 5 or more matching keys here and the scan
    // will have been canceled before it completed.
    console.log(matchingKeys);
});

Alternatively, we can use the eachScanCallback function parameter of the eachScan() method for more fine-grained cancellation of a scan.

Example

scanner.eachScan(scanPattern, (matchingKeys) => {
    // Do something arbitrary and then return `true` to cancel the scan.
    if (Math.random() > 0.5) {
        return true;
    }
}, (err, matchCount) => {
    if (err) throw(err);

    console.log(`Found ${matchCount} keys after canceling the scan arbitrarily.`);
});

Test

Tests are run via Istanbul and Mocha. Clone the project then run:

npm run test

Contribute

Simply open an issue or send a pull request. Not sure how to do that? Check out Github’s fast and free course on how to contribute to a project.

License

Licensed under the Apache License 2.0.