mongoarchive
v0.2.0
Published
Archive data in Amazon S3 from Mongo
Downloads
28
Readme
mongoarchive
Archive data in Amazon S3 from MongoDB.
IMPORTANT: BETA version.
Install and run
- npm install mongoarchive -g
- (optional) change user
- mongoarchive --config
- npm install pm2 -g
- Edit the file ~/.mongoarchive/conf.json and set your Amazon S3 and MongoDB security params
- mongoarchive --start
The conf.json file
- amazonS3: Object where configure the parameters of the Amazon S3 buckets. A key of this object is used in the collections property
- key
- secret
- region
- bucketName
- mongodbConnectionUri: a MongoDB connection URI https://docs.mongodb.com/manual/reference/connection-string/
- collections: MongoDB collections to archive in Amazon S3. Each one has 5 params:
- name: name of the collection
- offset: the process archives the data from the oldest to a number of days ago. The offset is this number of days
- field: collection date field to control the offset param (sure, you have something like 'createdAt' in you collection)
- s3: key of the amazonS3 object with the configuration the Amazon S3 bucket to use
- remove: control if you want to remove or not from MongoDB, the data archived in S3 from this collection
- tmpExportFilePath: File where mongoexport data is saved temporally
Mongoarchive creates a default configuration file for you with this data, but sure, you will need to edit it with your Amazon S3 and MongoDB security params.
How it works
Collection by collection and day by day:
- Read data from Mongo
- Upload it to Amazon S3 (look for the 'mongoarchive' folder in your bucket)
- Remove it from Mongo (optional. See the "remove" param in conf.json)
Before use it
- For now, this is a beta version. We are using it, but try first in a safe environment
- Create an index in the collection date field
Recover from S3 backups
- Download from Amazon S3
- Uncompress it
- mongoimport (example: mongoimport -h localhost:27017 -d db -c collection --file file)
Changelog
Releases are documented in the changelog file
Bugs
No bugs reported currently
TODOS
Confirmed
- Tests
Requests
- Add collection fields for each collection
- Save as csv and json files
- Automatic recover from S3 backups