@esmkit/node-mongotools

This project provides 2 wrappers :

• mongodump,
• mongorestore.

This project also include dropbox integration feature to

• dump and upload onto dropbox,
• restore from a dropbox hosted mongo backup.

There is an autonomous feature called rotation that provide a backup file rotation mechanism

• remove N oldest deprecated backups.

This readme contains some samples. Other samples are under examples/.

Command line usage

Initial setup - first time only

# get source code
git clone https://github.com/esmkit/node-mongotools.git
# install dependencies
npm install

#~ setup environment variables
cp env/initEnv.template.sh env/initEnv.dontpush.sh
# you must update env/initEnv.dontpush.sh

Set your preferences

# source your options
. ./env/initEnv.dontpush.sh

Basic feature

# create a mongo dump
node mt dump

# create a encrypted mongo dump
node mt dumpz

# list backups
node mt list

# restore a mongo local dump
# please note that mt restore use following options : dropBeforeRestore: true, deleteDumpAfterRestore: true
node mt restore backup/myDatabase__2020-11-08_150102.gz

# rotate backup files
node mt rotation

# Helper : show current options values
node mt options

Add in-line extra options

You could play with env default options plus in-line command extra options

# create dump of a given 'shippingprices' collection, provide a target backup name as '2023_01_shippingPrices.gz', and show mongodump command
MT_COLLECTION=shippingprices MT_FILENAME=2023_01_shippingPrices.gz MT_SHOW_COMMAND=true node mt dump

# show backup in list
node mt list

# using mongo: drop a given collection
mongo myDb --eval "db.shippingprices.drop()"

# restore collection
MSYS_NO_PATHCONV=1 MT_COLLECTION=shippingprices MT_SHOW_COMMAND=true node mt restore /backup/2023_01_shippingprices.gz
# Note that collection option will produce wildcard in nsInclude arg '--nsInclude myDb.*'

Dropbox feature

You have a dropbox access token in your preferences, (cf. "Mongo tools options")

# create a mongo dump is the same command
node mt dump
# restore a mongo dropbox dump
node mt restore /backup/myDatabase__2020-11-08_150102.gz

# git bash for windows users having absolute path issue could use the following command
unalias node
MSYS_NO_PATHCONV=1 node mt restore /backup/myDatabase__2020-11-08_150102.gz

# rotate local and dropbox backup files
node mt rotation

Library use

Install dependency

You have to import as dependency

npm install @esmkit/node-mongotools

Define the requirements, example:

import { MongoTools, MTOptions, MTCommand } from "@esmkit/node-mongotools";

var mongoTools = new MongoTools();
const mtOptions = {
        db: 'myDb',
        port: 17017,
        path: '/opt/myapp/backups',
        dropboxToken: process.env.MYAPP_DROPBOX_SECRET_TOKEN
      };

List dumps

var promiseResult = mongoTools.list(mtOptions);

Dump

var promiseResult = mongoTools.mongodump(mtOptions);

Restore

var promiseResult = mongoTools.mongorestore(mtOptions);

Rotation

var promiseResult = mongoTools.rotation(mtOptions);

Mongo tools options

Each mongotools feature rely on Mongo tools options (aka. MTOption).

Options precedence is the following:

• take options attribute if set,
• else take related environment variable if any,
• else take default value if any,
• else if it's a mandatory option, throw an error.

TIP: you could also show current options by doing:

console.log(new MTOptions());

shared options

These options are used by dump and restore.

Either uri or host/port/db:

| option | env | required | default value | description | | ------ | ------------ | -------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | uri | MT_MONGO_URI | true | (none) | mongodump uri, example mongodb+srv://granted-user:[email protected]/tMyDatababse. You could omit database name to dump all databases. |

or

| option | env | required | default value | description | | ---------- | ---------------- | -------- | ------------- | --------------------------------------------------------------------------------- | | db | MT_MONGO_DB | true | (none) | mongo database name. For dump only, you could set it to * to dump all databases | | host | MT_MONGO_HOST | false | 127.0.0.1 | mongo database hostname | | port | MT_MONGO_PORT | false | 27017 | mongo database port | | username | MT_MONGO_USER | false | (none) | mongo database username | | password | MT_MONGO_PWD | false | (none) | mongo database password | | authDb | MT_MONGO_AUTH_DB | false | admin | mongo auth database |

ssl options

Optional ssl related options

| option | env | required | default value | description | | ------------------- | ----------------------------- | -------- | ------------- | ---------------------------------------------------------------- | | ssl | MT_MONGO_SSL | false | (none) | if "1" then add --ssl option | | sslCAFile | MT_MONGO_SSL_CA_FILE | false | (none) | .pem file containing the root certificate chain | | sslPEMKeyFile | MT_MONGO_SSL_PEM_KEY_FILE | false | (none) | .pem file containing the certificate and key | | sslPEMKeyPassword | MT_MONGO_SSL_PEM_KEY_PASSWORD | false | (none) | password to decrypt the sslPEMKeyFile, if necessary | | sslCRLFile | MT_MONGO_SSL_CRL_FILE | false | (none) | pem file containing the certificate revocation list | | sslFIPSMode | MT_MONGO_SSL_FIPS | false | (none) | if "1" then use FIPS mode of the installed openssl library | | tlsInsecure | MT_MONGO_TLS_INSECURE | false | (none) | if "1" then bypass the validation for server's certificate chain |

mongodump options

| option | env | required | default value | description | | ------------------------ | ---------------------- | -------- | ----------------------- | --------------------------------------------------------------------- | | path | MT_PATH | false | backup | dump target directory, created if it doesn't exist | | dumpCmd | | false | mongodump | mongodump binary | | fileName | MT_FILENAME | false | <dbName_date_time.gz> | dump target filename | | encrypt | | false | false | encrypt the dump using secret | | secret | MT_SECRET | false | null | secret to use if encrypt is enabled | | encryptSuffix | | false | .enc | encrypt file suffix | | includeCollections | | false | (none) | Deprecated - please use collection | | collection | MT_COLLECTION | false | (none) | Collection to include, if not specified all collections are included | | excludeCollections | MT_EXCLUDE_COLLECTIONS | false | (none) | Collections to exclude, if not specified all collections are included | | numParallelCollections | | false | 4 | Number of collections mongodump should export in parallel. | | viewsAsCollections | | false | false | When specified, mongodump exports read-only views as collections. |

Simple example:

import { MongoTools, MTOptions, MTCommand } from "@esmkit/node-mongotools";
var mongoTools = new MongoTools();

mongoTools.mongodump({
   db:'myDatabase',
   path:'backup',
   username:'root', password:'mypass', authDb:'admin'
})
.then((success) => console.info("success", success) )
.catch((err) => console.error("error", err) );

mongorestore options

| option | env | required | default value | description | | ------------------------ | ------------ | -------- | -------------- | -------------------------------------------------------------- | | dumpFile | MT_DUMP_FILE | true | (none) | dump file to restore | | restoreCmd | | false | mongorestore | mongorestore binary | | dropBeforeRestore | | false | false | set it to true to append --drop option | | deleteDumpAfterRestore | | false | false | set it to true to remove restored backup file | | decrypt | | false | false | decrypt the dump using secret. Activated if suffix is detected | | secret | MT_SECRET | false | null | secret to use if decrypt is enabled |

Simple example:

import { MongoTools, MTOptions, MTCommand } from "@esmkit/node-mongotools";
var mongoTools = new MongoTools();

mongoTools.mongorestore({
   dumpFile:'backup/myDatabase__2020-11-8_160011.gz',
   username:'root', password:'mypass', authDb:'admin'
})
.then((success) => {
  console.info("success", success.message);
  if (success.stderr) {
    console.info("stderr:\n", success.stderr);// mongorestore binary write details on stderr
  }
})
.catch((err) => console.error("error", err) );

Dropbox options

You could create a dropbox app to get a token : cf. https://www.dropbox.com/developers/apps/ "Generated access token"

| option | env | required | default value | description | | ------------------ | --------------------- | -------- | ------------- | --------------------------------------- | | dropboxToken | MT_DROPBOX_TOKEN | false | (none) | activate dropbox feature if present | | dropboxLocalPath | MT_DROPBOX_LOCAL_PATH | false | "dropbox" | local directory to receive dropbox dump |

When a token is set,

• the list feature will list the / + path dropbox directory
• the mongodump feature will upload the dump onto / + path dropbox directory (in addition to spawn it locally),
• the mongorestore feature will use dumpFile as dropbox dump location and retrieve it into dropboxLocalPath before doing the mongorestore action.

Rotation options

A safe time windows is defined by :

• now - rotationWindowsDays day(s) ===> now
  where backups can't be removed.

Backup out of safe time windows are called deprecated backup.

• rotationMinCount: minimum deprecated backups to keep,
• rotationCleanCount: number of (oldest) deprecated backups to delete.

| option | env | required | default value | description | | --------------------- | ------------------------ | -------- | ------------- | ------------------------------------------------------ | | rotationDryMode | MT_ROTATION_DRY_MODE | false | false | dont do delete actions, just print it | | rotationWindowsDays | MT_ROTATION_WINDOWS_DAYS | true | 15 | safe time windows in days since now | | rotationMinCount | MT_ROTATION_MIN_COUNT | true | 2 | minimum deprecated backups to keep. | | rotationCleanCount | MT_ROTATION_CLEAN_COUNT | true | 10 | number of (oldest first) deprecated backups to delete. |

Simple example:

MT_ROTATION_CLEAN_COUNT=2 \
MT_ROTATION_DRY_MODE=true \
MT_ROTATION_WINDOWS_DAYS=3 \ node mt rotation

Example details: if there is a backup that is more than 3 days old, keep 2 newer ones and delete the 10 oldest.

Dropbox limits:

• rotation feature will not apply if dropbox backup target directory content contains more than 2000 files.