tjd-dataimport

Nodejs program that imports and normalizes U.S. Department of Labor Form 5500 data into a Mongodb

Installation

npm install tjd-dataimport

Usage

This program will connect to a mongodb instance, open a named file, and import the file into the mongodb database. It also performs some maintenance operations such as adding missing document properties and updating indices from legacy versions of the database. There are three main functions and they are driven by which command line arguments are provided.

1. UPDATE INDEXES. If --create_index is specified, indices are updated and the program exits. If this parameter is provided, the following parameter is processed: --dburl. All others are ignored.

2. UPDATE STRUCTURE. If --add_search_name is specified, the sponsor.searchName property is added to pension plan documents and the program exits. I run the program multiple times until it updates ZERO documents. If this parameter is provided, the following parameters are processed: --dburl --limit. All others are ignored.

3. IMPORT A FILE. If neither --create_index nor --add_search_name are specified, then the program will try to import, normalize, and store the employee benefit plans described in the named file.

This program accepts the following command line parameters:

--add_search_name

If specified, this parameter instructs the program to add the sponsor.searchName property to each pension plan document. (It skips insurance plans.)

Default

false

Example

node app.js --dburl=mongodb://mywebaddress/mymongodbname --add_search_name

--create_index

If specified, this parameter instructs the program to create indexes on the database and exit. No file is processed.

Default

false

Example

node app.js --dburl=mongodb://mywebaddress/mymongodbname --create_index

--dburl

Url of MongoDb to write records to.

Default

There is no meaningful default. This is a required argument.

Example

node app.js --input="DOL 5500.csv" --dburl=mongodb:://mywebaddress/mymongodbname

--duplicates

Flags whether to leave duplicate records along or replace them. If the value "keep" is provided, then existing records in the database are not overridden by newly read records.

Default

keep

Example

node app.js --input="DOL 5500.csv" --duplicates=keep

--input

Name of the input file to read. This file will be searched for in a "downloads" folder beneath the run folder. So that means that wherever you run this program from, you need to have a "downloads" folder in that directory.

Default

There is no default. This is a required argument.

Example

In this example, the program will look for ./downloads/DOL 5500.csv node app.js --input="DOL 5500.csv"

--limit

The maximum number of rows from the file to process. If a vaule of -1 is provided, the entire file is processed.

Default

-1

Example

node app.js --input="DOL 5500.csv" --limit=500

--log_skipped_records

Flags whether skipped records are logged to the console. This is useful for debugging. If this command line parameter is provided, then the program will log skipped records to the console.

Example

node app.js --input="DOL 5500.csv" --log_skipped_records

--skip

Number of records to skip from the beginning of the file. This is usefule when the file has to be processed in segments due to memory/heap constraints on host computer.

Default

0

Example

node app.js --input="DOL 5500.csv" --skip=100000

Example Invocation

The following command line

node app.js --input="DOL 5500.csv" --limit=100 --duplicates=skip

Will produce output like this:

Connecting to MondoDb
Connected to MongoDb OK.
Promise { <pending> }
Promise { <pending> }
Promise { <pending> }
Processing DOL 5500.csv (Mon Jul 25 2016 00:06:40 GMT-0500 (Central Daylight Time))
---------------------------------------------------
Unfiltered plan count 101
Filtered plan count 101
pause pendingDbWrites: 101 recordsWritten: 0
Exit pending pendingDbWrites: 101 recordsWritten: 0
Exit pending pendingDbWrites: 8 recordsWritten: 93
Done. Processed 100 records

Notes

I can't imagine that application has any general use. It is specifically designed to import a list of employee benefit plans and import them into a Mongo database for searching by various apps.

To Do

The program works.

Copyright

Copyright (c) 2016 by Thomas J. Daley <[email protected]>