Couchit

Couchit is a database iterator with tools to validate and manage documents in a CouchDB database.

Couchit runs a set of user-defined JavaScript functions against all documents in a CouchDB database or view, or only some of them by specifying a start and/or an end key(s). Couchit comes with a built-in set of Utility Functions to perform helpful operations such as generating hashes, performing json-schema validation, and storing objects from iteration for post-processing.

Installation

Command Line Interface (CLI)

Couchit can run as a stand-alone application. Install it via npm:

npm install -g couchit

Then, to run:

couchit ./config.js

As a module

Couchit can also run as a module in another program. First, add it to your projects package.json

npm install --save couchit

Then, include it in your program:

const Couchit = require('couchit');
const config = require('./config.js');
new Couchit().iterate(config, console.log);

Configuration

Couchit can be controlled by environment variables or a config file.

The preferred way is through environment variables. This allows Couchit to run without storing sensititve information in code. It will be useful when setting Couchit to run as an AWS Lambda, for example.

Configuration Settings

The following list all configuration settings. If no environment variables are set, the Default Value will be used.

| Setting | Description | Default Value | | ---------------- |-------------| ---------------- | | COUCHDB_ENDPOINT | URI and port of CouchDB server, (does not include http://) | 'localhost:5984' | | COUCHDB_DATABASE | CouchDB database | 'db' | | COUCHDB_USERNAME | CouchDB user | 'couchdb' | | COUCHDB_PASSWORD | CouchDB password | 'couchdb' | | OPTS_INTERVAL | Number of ms to wait between page requests | 100 | | OPTS_START_KEY | View start key | null | | OPTS_END_KEY | View end key | null | | OPTS_PAGE_SIZE | Number of documents to retrieve per batch | 1000 | | OPTS_NUM_PAGES | Number of pages to retrieve | undefined | | OPTS_BATCH_SIZE | Batch update size | 100 | | OPTS_QUIET | Suppress report at end of run | false | | OPTS_TASKS | JavaScript functions with tasks to run | { "count-docs": (util, doc) => { util.count('total-docs') } } |

You can use a combination of environment variables and config.js Default Values to run Couchit. Just remember that environment variables always override defaults.

Setting an environment variable in Windows Powershell

$env:COUCHDB_PASSWORD="couchdb"

Setting an environment variable in macOS/Linux bash

export COUCHDB_PASSWORD="couchdb"

Tasks

Tasks are how you validate and manage documents as they are iterated over. There are a number of built-in Utility Functions that can be used by calling their util method, for example, to get a document hash: util.hash(doc). Additional document functionality is provided via util.nano, which exposes nano document functions. See the dependent-updates task below for an example of how nano document functions can be used.

Utility Functions

On each document iteration, the following functions are available via the Util() object:

| Function | Description | | ---------- | ------------- | | audit | Add an object to the audit array, which is returned in the callback | | count | Increment a counter associated with a particular key | | dereference | Reference external json-schema file definitions from other files | | log | Alias for console.log | | hash | Generate a SHA256 hash for a given document, object, or string | | nano | Exposes nano document functions | remove | Delete the document from the database | | save | Save the document back to the database | | validate | Validate a document using json-schema | | incrementWaits | Set a wait. Used to ensure asyncronous process can complete for callback response | | decrementWaits | Remove a wait |

Example Tasks

You can define any number of named tasks to run, which will be run once per document. The following are examples of common tasks that may be useful to adopt for your needs.

Count all documents:

This is a trivial example where a counter (total-docs) is incremented for each document retrieved.

{
    "count-docs": (util, doc) => util.count('total-docs'),
}

Audit data for later user:

By using util.audit(), you can store any object for use after processing has completed. This is useful for initiating a post-processing step that is based on the output of Couchit run.

{
    "audit-bad-docs": (util, doc) => {
        if (doc.status && doc.status === 'bad') {
            const object = { bad_doc_id: doc._id, status: doc.status }
            util.audit(object);
        }
    }
}

Calculate the hash of part of a document:

By using util.audit(), you can store any object for use after processing has completed. This is useful for initiating a post-processing step that is based on the output of Couchit run.

{
    "hash-doc-contents": (util, doc) => {
        const hash = util.hash(doc);
        util.log('SHA256 hash of doc: ' + hash);
    }
}

Validate a schema:

Determine if a document is valid based on a json-schema specification. This task uses ajv for validation.

{
    "validate-schema": (util, doc) => {
        const schema = require('./test/schema.json');
        const data = require('./test/data.json');
        const valid = util.validate(schema, data);
        util.log(doc._id + ' is valid? ' + valid);
    }
}

Update a parent document by checking for existence of child docs:

Note that nano functions run asyncrounously while documents are interated over. To ensure complete stats and (optional) audit object tracking, use the util.incrementWaits() to set a wait prior calling the async function and util.decrementWaits() to remove the wait upon completion.

{
    "dependent-updates": (util, doc) => {
      if (doc.childKeys) {
        const keys = doc.childKeys;

        // Set a wait
        util.incrementWaits();

        // Do a bulk get for all childKeys
        util.nano.fetch(keys, (err, result) => {
          if (err) {
            console.log(err);
          } else {
            const newChildKeys = [];
            const rows = result.rows;
            const hasMissingChildKeys = false;

            // Only add docs to the newChildKeys that were found in the db
            rows.forEach(row => {
              if (row.doc) {
                newChildKeys.push(row.id);
              } else {
                hasOrphans = true;
                util.log('Missing child doc: ' + row.id);
              }
            });

            if (hasMissingChildKeys) {
              doc.childKeys = newChildKeys;
              util.nano.insert(doc, (err, result) => {
                console.log(result);

                // Remove a wait
                util.decrementWaits();
              });
            }
          }
        });
      }
    }
}

Credits

• Couchit is based on Couchtato; thanks to Cliffano Subagio for his work.