node-mongotools
v2.2.6
Published
node-mongotools: a mongotools wrapper for nodejs. Freely inspired from github.com/amulyakashyap09/mbr
Downloads
901
Maintainers
Readme
node-mongotools
This project provides 2 wrappers for :
- mongodump,
- mongorestore.
This project also include dropbox integration feature to
- dump and upload onto dropbox,
- restore from a dropbox hosted mongo backup.
There is an autonomous feature called rotation that provide a backup file rotation mechanism
- remove N oldest deprecated backups.
This readme contains some samples. Other samples are under examples/.
You may notice that mongodump
and mongorestore
binaries are part of database-tools (follow standard installation for common OS). These binaries are mandatory for node-mongotools nodejs project as we are talking here of "wrapper" : a more convenient way to launch command + arguments from NodeJS code).
From a GitHub Actions context, you can install them via action-mongo-tools. There is an example in this project in main workflow that execute tests.
Command line usage
You should decide to install it globally without getting source or by cloning repository.
Install globally (beta)
npm install -g node-mongotools@latest
When using a global install node-mongotools
command must be directly available (ie. you can replace node mt
by node-mongotools
in examples).
You could revert thins by using
npm uninstall -g node-mongotools
Install by getting source
git clone https://github.com/boly38/node-mongotools.git
# install dependencies
npm install
Environment setup - first time only
#~ setup environment variables
cp env/initEnv.template.sh env/initEnv.dontpush.sh
# you must update env/initEnv.dontpush.sh
Set your preferences
source your options
. ./env/initEnv.dontpush.sh
Basic feature
# create a mongo dump
node mt dump
# or if you rely on a global install
# node-mongotools dump
# apply the same logic for next examples
# create a encrypted mongo dump
node mt dumpz
# list backups
node mt list
# restore a mongo local dump
# please note that mt restore use following options : dropBeforeRestore: true, deleteDumpAfterRestore: true
node mt restore backup/myDatabase__2020-11-08_150102.gz
# rotate backup files
node mt rotation
# Helper : show current options values
node mt options
Add in-line extra options
You could play with env default options plus in-line command extra options
# create dump of a given 'shippingprices' collection, provide a target backup name as '2023_01_shippingPrices.gz', and show mongodump command
MT_COLLECTION=shippingprices MT_FILENAME=2023_01_shippingPrices.gz MT_SHOW_COMMAND=true node mt dump
# show backup in list
node mt list
# using mongo: drop a given collection
mongo myDb --eval "db.shippingprices.drop()"
# restore collection
MSYS_NO_PATHCONV=1 MT_COLLECTION=shippingprices MT_SHOW_COMMAND=true node mt restore /backup/2023_01_shippingprices.gz
# Note that collection option will produce wildcard in nsInclude arg '--nsInclude myDb.*'
Dropbox feature
You have a dropbox access token in your preferences, (cf. "Mongo tools options")
# create a mongo dump is the same command
node mt dump
# restore a mongo dropbox dump
node mt restore /backup/myDatabase__2020-11-08_150102.gz
# git bash for windows users having absolute path issue could use the following command
unalias node
MSYS_NO_PATHCONV=1 node mt restore /backup/myDatabase__2020-11-08_150102.gz
# rotate local and dropbox backup files
node mt rotation
Library use
Install dependency
You have to import as dependency
npm install node-mongotools
Define the requirements, example:
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();
const mtOptions = {
db: 'myDb',
port: 17017,
path: '/opt/myapp/backups',
dropboxToken: process.env.MYAPP_DROPBOX_SECRET_TOKEN
};
This project is compatible with ES Module projects that rely on import
.
For now it's not compatible with CommonJS (require
). But contribution are welcome.
List dumps
var promiseResult = mongoTools.list(mtOptions);
Dump
var promiseResult = mongoTools.mongodump(mtOptions);
Restore
var promiseResult = mongoTools.mongorestore(mtOptions);
Rotation
var promiseResult = mongoTools.rotation(mtOptions);
Mongo tools options
Each mongotools feature rely on Mongo tools options (aka. MTOption).
Options precedence is the following:
- take
options
attribute if set, - else take related environment variable if any,
- else take default value if any,
- else if it's a mandatory option, throw an error.
TIP: you could also show current options by doing:
console.log(new MTOptions());
shared options
These options are used by dump and restore.
Either uri
or host
/port
/db
:
| option | env | required | default value | description |
|--------|--------------|----------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| uri
| MT_MONGO_URI
| true | (none) | mongodump uri, example mongodb+srv://granted-user:[email protected]/tMyDatababse
. You could omit database name to dump all databases. |
or
| option | env | required | default value | description |
|------------|------------------|----------|---------------|-------------------------------------------------------------------------------------|
| db
| MT_MONGO_DB
| true | (none) | mongo database name. For dump only, you could set it to *
to dump all databases |
| host
| MT_MONGO_HOST
| false | 127.0.0.1
| mongo database hostname |
| port
| MT_MONGO_PORT
| false | 27017
| mongo database port |
| username
| MT_MONGO_USER
| false | (none) | mongo database username |
| password
| MT_MONGO_PWD
| false | (none) | mongo database password |
| authDb
| MT_MONGO_AUTH_DB
| false | admin
| mongo auth database |
ssl options
Optional ssl related options
| option | env | required | default value | description |
|----------------------|-------------------------------|----------|----------------|-------------------------------------------------------------------|
| ssl
| MT_MONGO_SSL
| false | (none) | if "1" then add --ssl
option |
| sslCAFile
| MT_MONGO_SSL_CA_FILE
| false | (none) | .pem file containing the root certificate chain |
| sslPEMKeyFile
| MT_MONGO_SSL_PEM_KEY_FILE
| false | (none) | .pem file containing the certificate and key |
| sslPEMKeyPassword
| MT_MONGO_SSL_PEM_KEY_PASSWORD
| false | (none) | password to decrypt the sslPEMKeyFile, if necessary |
| sslCRLFile
| MT_MONGO_SSL_CRL_FILE
| false | (none) | pem file containing the certificate revocation list |
| sslFIPSMode
| MT_MONGO_SSL_FIPS
| false | (none) | if "1" then use FIPS mode of the installed openssl library |
| tlsInsecure
| MT_MONGO_TLS_INSECURE
| false | (none) | if "1" then bypass the validation for server's certificate chain |
mongodump options
| option | env | required | default value | description |
|--------------------------|------------------------|----------|-------------------------|------------------------------------------------------------------------------|
| path
| MT_PATH
| false | backup
| dump target directory, created if it doesn't exist |
| dumpCmd
| MT_MONGODUMP
| false | mongodump
| mongodump binary |
| fileName
| MT_FILENAME
| false | <dbName_date_time.gz>
| dump target filename |
| encrypt
| | false | false | encrypt the dump using secret |
| secret
| MT_SECRET
| false | null | secret to use if encrypt is enabled (aes-256-ctr require 32 byte length key) |
| encryptSuffix
| | false | .enc
| encrypt file suffix |
| includeCollections
| | false | (none) | Deprecated - please use collection
|
| collection
| MT_COLLECTION
| false | (none) | Collection to include, if not specified all collections are included |
| excludeCollections
| MT_EXCLUDE_COLLECTIONS
| false | (none) | Collections to exclude, if not specified all collections are included |
| numParallelCollections
| | false | 4 | Number of collections mongodump should export in parallel. |
| viewsAsCollections
| | false | false | When specified, mongodump exports read-only views as collections. |
Simple example:
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();
mongoTools.mongodump({
db:'myDatabase',
path:'backup',
username:'root', password:'mypass', authDb:'admin'
})
.then((success) => console.info("success", success) )
.catch((err) => console.error("error", err) );
mongorestore options
| option | env | required | default value | description |
|--------------------------|------------------|----------|----------------|----------------------------------------------------------------|
| dbFrom
| MT_MONGO_DB_FROM
| false | (none) | name of the source db (if db
is not specified)
| dbTo
| MT_MONGO_DB_TO
| false | (none) | name of the target db (if db
is not specified) |
| dumpFile
| MT_DUMP_FILE
| true | (none) | dump file to restore |
| restoreCmd
| MT_MONGORESTORE
| false | mongorestore
| mongorestore binary |
| dropBeforeRestore
| | false | false | set it to true
to append --drop
option |
| deleteDumpAfterRestore
| | false | false | set it to true
to remove restored backup file |
| decrypt
| | false | false | decrypt the dump using secret. Activated if suffix is detected |
| secret
| MT_SECRET
| false | null | secret to use if decrypt is enabled |
Simple example:
import { MongoTools, MTOptions, MTCommand } from "node-mongotools";
var mongoTools = new MongoTools();
mongoTools.mongorestore({
dumpFile:'backup/myDatabase__2020-11-8_160011.gz',
username:'root', password:'mypass', authDb:'admin'
})
.then((success) => {
console.info("success", success.message);
if (success.stderr) {
console.info("stderr:\n", success.stderr);// mongorestore binary write details on stderr
}
})
.catch((err) => console.error("error", err) );
Dropbox options
You could create a dropbox app to get a token : cf. https://www.dropbox.com/developers/apps/ "Generated access token"
| option | env | required | default value | description |
|-----------------------|----------------------------|-----------|-----------------|----------------------------------------------------------------------|
| dropboxLocalPath
| MT_DROPBOX_LOCAL_PATH
| false | "dropbox" | local directory to receive dropbox dump |
| dropboxAppKey
| MT_DROPBOX_APP_KEY
| false | (none) | (refresh token based) dropbox feature application key. (*1) |
| dropboxAppSecret
| MT_DROPBOX_APP_SECRET
| false | (none) | (refresh token based) dropbox feature application secret. (*1) |
| dropboxRefreshToken
| MT_DROPBOX_REFRESH_TOKEN
| false | (none) | (refresh token based) dropbox feature application refreshToken. (*1) |
| dropboxToken
(*2) | MT_DROPBOX_TOKEN
| false | (none) | DEPRECATED - activate dropbox feature if present. (*2) |
(*1) dropboxAppKey
, dropboxAppSecret
and dropboxRefreshToken
(long-lived offline refresh token) are required to activate optional dropbox feature. You have some help about how to get them in dropbox-refresh-token dedicated helper repository.
(*2) dropboxToken
option is DEPRECATED : This was legacy old-long-lived access-token - this kind of token are no more available from dropbox developers portal (src). Please switch to dropboxAppKey
, dropboxAppSecret
, and dropboxRefreshToken
instead. For now this is still supported.
When a token is set,
- the
list
feature will list the/
+path
dropbox directory - the
mongodump
feature will upload the dump onto/
+path
dropbox directory (in addition to spawn it locally), - the
mongorestore
feature will usedumpFile
as dropbox dump location and retrieve it intodropboxLocalPath
before doing the mongorestore action.
Rotation options
A safe time windows is defined by :
now - rotationWindowsDays day(s)
===>now
where backups can't be removed.
Backup out of safe time windows are called deprecated backup
.
rotationMinCount
: minimum deprecated backups to keep,rotationCleanCount
: number of (oldest) deprecated backups to delete.
| option | env | required | default value | description |
|-----------------------|--------------------------|----------|---------------|--------------------------------------------------------|
| rotationDryMode
| MT_ROTATION_DRY_MODE
| false | false | dont do delete actions, just print it |
| rotationWindowsDays
| MT_ROTATION_WINDOWS_DAYS
| true | 15 | safe time windows in days since now |
| rotationMinCount
| MT_ROTATION_MIN_COUNT
| true | 2 | minimum deprecated backups to keep. |
| rotationCleanCount
| MT_ROTATION_CLEAN_COUNT
| true | 10 | number of (oldest first) deprecated backups to delete. |
Simple example:
MT_ROTATION_CLEAN_COUNT=2 \
MT_ROTATION_DRY_MODE=true \
MT_ROTATION_WINDOWS_DAYS=3 \ node mt rotation
Example details: if there is a backup that is more than 3 days old, keep 2 newer ones and delete the 10 oldest.
Dropbox limits:
- rotation feature will not apply if dropbox backup target directory content contains more than 2000 files.
How to contribute
You're not a dev ? just submit an issue (bug, improvements, questions). Or else:
- Clone
- Install deps
- Then mocha tests
git clone https://github.com/boly38/node-mongotools.git
npm install
npm run test
- you could also fork, feature branch, then submit a pull request.
Services or activated bots
| badge | name | description |
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|:----------------------------------------------------------|
| | Github actions | Continuous tests. |
| | Github actions | Continuous vulnerability audit. |
| | Houndci | JavaScript automated review (configured by .hound.yml
) |
Contributions
provided by Repobeats