refilter

Containerizable utility to filter JSON collections in Redis.

Use case

• use mongoexport to export a collection from MongoDB into a file
• stream each line into a Redis list using https://github.com/evanx/resplit
• reimport pops each line, extracts its ID field for the Redis key, and sets the JSON document in Redis

Config spec

See lib/spec.js https://github.com/evanx/refilter/blob/master/lib/spec.js

module.exports = {
}

Appication archetype

Incidently lib/index.js uses the redis-app-rpf application archetype.

require('redis-app-rpf')(require('./spec'), require('./main'));

where we extract the config from process.env according to the spec and invoke our main function.

This provides lifecycle boilerplate reused across similar applications.

See https://github.com/evanx/redis-app-rpf

Docker

You can build as follows:

docker build -t refilter https://github.com/evanx/refilter.git

using https://github.com/evanx/refilter/blob/master/Dockerfile

FROM node:7.5.0
ADD package.json .
RUN npm install
ADD lib lib
ENV NODE_ENV production
CMD ["node", "--harmony", "lib/index.js"]

See test/demo.sh https://github.com/evanx/refilter/blob/master/test/demo.sh

Builds:

• isolated network refilter-network
• isolated Redis instance named refilter-redis
• this utility as refilter-instance

Isolated test network

First we create the isolated network:

docker network create -d bridge refilter-network

Disposable Redis instance

Then the Redis container on that network:

redisContainer=`docker run --network=refilter-network \
    --name $redisName -d redis`
redisHost=`docker inspect $redisContainer |
    grep '"IPAddress":' | tail -1 | sed 's/.*"\([0-9\.]*\)",/\1/'`

where we parse its IP number into redisHost

Setup test data

Build and run

We build a container image for this service:

docker build -t refilter https://github.com/evanx/refilter.git

We interactively run the service on our test Redis container:

docker run --name refilter-instance --rm -i \
  --network=refilter-network \
  -e redisHost=$redisHost \
  refilter

Verify results

We check the lengths of the various queues:

redis-cli -h $redisHost llen resplit:q |
  grep ^0$

We check that the key is pushed to the output queue:

+ redis-cli -h 172.27.0.2 lindex refile:key:q 0
place:ChIJV3iUI-PPdkgRGA7v4bhZPlU:j

evan@dijkstra:~/refilter$ sh test/demo.sh
...

Teardown

docker rm -f refilter-redis
docker network rm refilter-network

Implementation

See lib/main.js

while (true) {
    logger.debug('brpoplpush', config.inq, config.busyq, config.popTimeout);
    const item = await client.brpoplpushAsync(config.inq, config.busyq, config.popTimeout);
    logger.debug('popped', config.inq, config.busyq, item);
    if (!item) {
        break;
    }
    if (item === 'exit') {
        await client.lrem(config.busyq, 1, item);
        break;
    }
}

Appication archetype

Incidently lib/index.js uses the redis-app-rpf application archetype.

require('redis-app-rpf')(require('./spec'), require('./main'));

where we extract the config from process.env according to the spec and invoke our main function.

This provides lifecycle boilerplate to reuse across similar applications.

See https://github.com/evanx/redis-app-rpf.