mgeneratejs

mgeneratejs generates structured, semi-random JSON data according to a template object. It offers both a command line script and a JavaScript API.

Installation

npm install -g @econsult/mgeneratejs

Install and dev locally

// to build and install the package:

npm pack
npm install --global econsult-mgeneratejs-<package_version>.tgz


npm run fmt // fixes formatting violations

npm run ci // checks formatting and runs unit tests

Example

mgeneratejs '{"name": "$name", "age": "$age", "emails": {"$array": {"of": "$email", "number": 3}}}' -n 5

Results in:

{"name":"Glenn Simmons","age":32,"emails":["[email protected]","[email protected]","[email protected]"]}
{"name":"Jane Santiago","age":57,"emails":["[email protected]","[email protected]","[email protected]"]}
{"name":"Winifred Martinez","age":59,"emails":["[email protected]","[email protected]","[email protected]"]}
{"name":"Helena Chandler","age":65,"emails":["[email protected]","[email protected]","[email protected]"]}
{"name":"Gary Allison","age":30,"emails":["[email protected]","[email protected]","[email protected]"]}

You can also specify a JSON file instead of a JSON string:

mgeneratejs template.json -n 5

And pipe it to Mongo using mongoimport:

mgeneratejs mongotemplate.json -c mongotemplatecontext.json -n 10 | mongoimport -c myCollection -d myDatabase

Template Syntax

The input string or file must be valid JSON, with one exception: it's ok to omit quotes around keys, so both these templates are equivalent:

{"name": "$name"}
{name: "$name"}

Shape of objects

The output has the same shape as the input template (including nested keys), with one exception: If a key is assigned the special value $missing, then the key is not present in the output (see $missing below for an example).

Values

All values are taken literally, except for special $-prefixed values. These values are called "operators". A list of operators can be found below.

Operators are used either in string or object format. The string format is a shortcut to call the operator with default options.

String format:

{"key": "$operator"}

Object format:

{"key": {"$operator": { <additional options> }}}

Most operators have sensible default values that are used for their string format.

Example: $year

mgeneratejs '{"born_in": "$year"}' -n 5

{"born_in":"2035"}
{"born_in":"2086"}
{"born_in":"2088"}
{"born_in":"2022"}
{"born_in":"2082"}

The object format allows to pass in additional options to the operator, here, a minimum and maximum for the value:

mgeneratejs '{"born_in": {"$year": {"min": 1930, "max": 1970}}}'

{"born_in":"1936"}
{"born_in":"1953"}
{"born_in":"1964"}
{"born_in":"1932"}
{"born_in":"1943"}

See the definition of the operator for its default values.

Combining Operators

Operators can be combined, where the result of one operator is passed in as an option to another operator.

Example: Here we pass in a random number between 0 and 5 to the number option of the $array operator to generate variable-length arrays.

mgeneratejs '{"ip_addresses": {"$array": {"of": "$ip", "number": {"$integer": {"min": 0, "max": 5}}}}}'

{"ip_addresses":["166.182.72.83","127.94.56.191","236.79.131.157","94.66.121.242"]}
{"ip_addresses":["48.227.145.186","160.173.45.84","24.86.124.235"]}
{"ip_addresses":[]}
{"ip_addresses":["21.45.212.198"]}
{"ip_addresses":["199.209.162.241"]}

Built-in Operators

General

• $array: Creates an array of values.
• $choose: Chooses one element from an array of possible choices.
• $inc: Generates natural numbers in increasing order.
• $join: Joins elements of an array to a string.
• $pick: Returns an element from an array.
• $pickset: Returns a subset of an array.

Geospatial

• $address: Returns a full address.
• $coordinates: Returns a pair of longitude/latitude coordinates.
• $point: Returns a GeoJSON Point.
• $linestring: Returns a GeoJSON LineString.
• $polygon: Returns a GeoJSON Polygon.
• $geometries: Returns a GeoJSON GeometryCollection.

Native and MongoDB-specific Types

• $binary: Returns a MongoDB Binary type.
• $date: Returns a random date, optionally in a given range.
• $now: Returns the current date.
• $maxkey: Returns a MongoDB MaxKey object.
• $minkey: Returns a MongoDB MinKey object.
• $numberDecimal: Returns a MongoDB Decimal128 number.
• $numberLong: Returns a MongoDB Long (Int64) number.
• $numberInt: Returns a Int32 number.
• $objectid: Returns MongoDB ObjectID.
• $regex: Returns a Regular Expression object.
• $timestamp: Returns a MongoDB Timestamp.

All Built-in Operators in Alphabetical Order

$address

Creates a full address for a random locale. The address may not necessarily make factual sense (e.g. country of Bulgaria but with a UK postcode).

Example

{"address": "$address"}

Returns a full locale-driven address, e.g. {"address":"石中心62510号 Suite 371, 安海市, United States of America 496422"}

$age

Calculates the age in years based on the supplied date of birth in the form YYYY-MM-DD.

Options

• dob (required) the date of birth.

Example

{"ageInYears": {"$age": {"dob": "1999-11-21"}}}

Returns {"ageInYears": "22"} (as of 2022-08-18)

$array

Creates an array of values. Each new element is evaluated separately.

Options

• of (required) Defines an element of the array. Operators are evaluated separately for each element.
• number (optional) Number of elements. Default 0.

Example

{"countries": {"$array": {"of": {"$country": {"full": true}}, "number": 3}}}

Creates an array of 3 countries, e.g. {"countries":["Czech Republic","Ireland","Argentina"]}

$binary

Returns a random MongoDB Binary value, optionally with a length and subtype.

Options

• length (optional) Length in bytes of binary value. Default 10.
• subtype (optional) Specific binary subtype (see BSON spec). Default 0.

Example

{"blob": "$binary"}

Returns a Binary object (stringified to extended JSON on stdout). e.g. {"blob":{"$binary":"TzhXcFZoRllRNg==","$type":"0"}}.

$choose

Chooses one element from an array of possible choices with uniform probability. Optionally chooses with probability proportional to a provided weights array.

Options

• from (required) Array of values or operators to choose from.
• weights (optional) Number of elements. Default 0.

Example

{"status": {"$choose": {"from": ["read", "unread", "deleted"], "weights": [2, 1, 1]}}}

Returns {"status": "read"} with probability 1/2, and {"status": "unread"} and {"status": "deleted"} each with probability 1/4.

$coordinates

Returns a 2-element array of longitude/latitude coordinates, optionally within long_lim and/or lat_lim bounds.

Aliases

• $coord
• $coordinate

Options

• long_lim (optional) Array of longitude bounds. Default [-180, 180].
• lat_lim (optional) Array of latitude bounds. Default [-90, 90].

Example

{"position": {"$coordinates": {"long_lim": [-20, -19]}}}

Returns a pair of coordinates with the longitude bounds between -20 and -19, e.g. {"position":[-19.96851,-47.46141]}.

$date

Returns a random date object, optionally between specified min and max values. If min and/or max are provided, they need to be in a format that Date.parse() can read, e.g. ISO-8601.

Aliases

• $datetime

Options

• min (optional) Minimum date, as parseable string.
• max (optional) Maximum date, as parsable string.

Example

{"last_login": {"$date": {"min": "2015-01-01", "max": "2016-12-31T23:59:59.999Z"}}}

Returns a random date and time between 2015 and 2016 (incl.), e.g. {"last_login":{"$date":"2016-06-28T15:28:54.721Z"}}.

$now

Returns the current date at creation time. Ideal for time-stamping documents.

Options none

Example

{"created": "$now"}

Returns the extended JSON date and time at creation. {"created":{"$date":"2017-02-20T04:44:24.880Z"}}.

$geometries

Returns a GeoJSON formatted GeometryCollection with number geometries. By default, the geometries are chosen from Point, LineString, and Polygon. A subset of types can be specified with the types option.

Additional options are passed onto each geometry, e.g. corners is passed to polygons, locs is passed to line strings.

Options

• number (optional) Number of geometries in the collection. Default 3.
• types (optional) Types of geometries to choose from. Default ["Point", "LineString", "Polygon"].
• locs (optional) Number of locations in a line string. Default 2.
• corners (optional) Number of corners in a polygon. Default 3. The last point in the coordinates array closes the polygon and does not count towards the number of corners.
• long_lim (optional) Array of longitude bounds. Default [-180, 180].
• lat_lim (optional) Array of latitude bounds. Default [-90, 90].

Example

{"triangles": {"$geometries": {"types": ["Polygon"], "corners": 3, "number": 4}}}

Returns a GeoJSON GeometryCollection with 4 triangles.

{
  "triangles": {
    "type": "GeometryCollection",
    "geometries": [
      {
        "coordinates": [[[39.3259,-16.71813],[172.02089,-14.75681],[61.97122,-1.4036],[39.3259,-16.71813]]],
        "type": "Polygon"
      },
      {
        "coordinates": [[[57.66865,-18.3085],[-48.81722,-40.64912],[-145.11102,32.8189],[57.66865,-18.3085]]],
        "type": "Polygon"
      },
      {
        "coordinates": [[[110.68379,28.31158],[-73.67573,-19.54736],[-73.29514,52.07583],[110.68379,28.31158]]],
        "type": "Polygon"
      },
      {
        "coordinates": [[[-29.36382,79.19853],[138.84298,7.43148],[176.28313,36.83292],[-29.36382,79.19853]]],
        "type": "Polygon"
      }
    ]
  }
}

$inc

Generate natural numbers in increasing order.

Options

• start (optional) starts counting at this value. Default 0.
• step (optional) increases by this amount each time. Default 1. Can also take negative value.

Example

{"even_numbers": {"$inc": {"start": 0, "step": 2}}}

Assigns the numbers 0, 2, 4, 6, ... to subsequent objects.

$indexOf

Finds the index of the given value in the given list.

Options

• from (mandatory) the array list to find the value in
• value (mandatory) the value to lookup

Example

{
  "derivedIndex": {
    "$indexOf": {
      "from": ["vanilla", "chocolate", "lemon"],
      "value": "chocolate"
    }
  }
}

Returns {"derivedIndex": 1}

$join

Takes an array array and a separator string sep and joins the elements of the array (each cast to string) separated by sep. The default separator is the empty string ''.

Options

• array (required) Array of values to be joined (cast to string).
• sep (optional) Separator string. Default '' (empty string).

Example

{"code": {"$join": {"array": ["foo", "bar", "baz"], "sep": "-"}}}

Returns {"code": "foo-bar-baz"}.

$linestring

Returns a GeoJSON formatted LineString with optionally locs locations and within long_lim and/or lat_lim bounds.

Options

• locs (optional) Number of locations in the line string. Default 2.
• long_lim (optional) Array of longitude bounds. Default [-180, 180].
• lat_lim (optional) Array of latitude bounds. Default [-90, 90].

Example

{"line": "$linestring"}

Returns a GeoJSON line string with 2 locations, e.g. {"line":{"type":"LineString","coordinates":[[35.67106,-41.9745],[120.07739,68.46491]]}}.

$maxkey

Returns the MongoDB MaxKey value.

Example

{"upper_bound": "$maxkey"}

Returns {"upper_bound":{"$maxKey":1}}.

$minkey

Returns the MongoDB MinKey value.

Example

{"lower_bound": "$minkey"}

Returns {"lower_bound":{"$minKey":1}}.

$numberDecimal

Returns a MongoDB Decimal128 number.

Aliases

• $decimal

Options

• min (optional) minimum value. Default 0.
• max (optional) maximum value. Default 1000.
• fixed (optional) number of digits after the decimal. Default 2.

Example

{"price": {"$numberDecimal": {"fixed": 3}}}

Returns {"price":{"$numberDecimal": "1545.241"}}.

$numberLong

Returns a MongoDB Long (Int64) number.

Aliases

• $long

Options

• min (optional) minimum value. Default -2^53.
• max (optional) maximum value. Default 2^53.

Example

{"price": {"$numberLong": {"min": 100000}}}

Returns {"price":{"$numberLong":"7624790980443125"}}.

$numberInt

Returns ag 32-bit integer number.

Aliases

• $number
• $integer

Options

• min (optional) minimum value. Default -2^31.
• max (optional) maximum value. Default 2^31.

Example

{"price": {"$numberLong": {"min": 100000}}}

Returns {"price":{"$numberLong":"7624790980443125"}}.

$objectid

Returns a new MongoDB ObjectId.

Aliases

• $oid

Example

{"_id": "$objectid"}

Returns {"_id":{"$oid":"574ac75f725f4447309ab587"}}.

$pick

Takes an array and a number element and returns the element-th value of the array. If the number is larger than the length of the array, return $missing instead, which will remove the key from the resulting document. element is zero-based (0 returns the first element).

Options

• array (required) Array of values or operators to choose from.
• element (optional) Index of the array element to pick. Default 0.

Example

 {"color": {"$pick": {"array": ["green", "red", "blue"], "element": 1}}}

Returns {"color": "red"}.

$pickset

Takes an array and a number quantity and returns a new n-element array containing unique values from the input array. If the number is larger than the length of the array, return $missing instead, which will remove the key from the resulting document.

Options

• array (required) Array of values or operators to choose from.
• quantity (optional) The size of the output array. Default 1.

Example

{"color": {"$pickset": {"array": ["green", "red", "blue"], "quantity": 2}}}

Returns {"color": ["red", "green"]}

$point

Like $coordinates, but returns a GeoJSON formatted Point, optionally within long_lim and/or lat_lim bounds.

Options

• long_lim (optional) Array of longitude bounds. Default [-180, 180].
• lat_lim (optional) Array of latitude bounds. Default [-90, 90].

Example

{"position": {"$point": {"long_lim": [-20, -19]}}}

Returns a GeoJSON Point with the longitude bounds between -20 and -19, e.g. {"position": {"type": "Point", "coordinates": [-19.96851,-47.46141]}}.

linestring: require('./linestring'), polygon: require('./polygon'), geometries: require('./geometries'),

$polygon

Returns a GeoJSON formatted Polygon (without holes) with corners corners, optionally within long_lim and/or lat_lim bounds. The last point in the coordinates array closes the polygon and does not count towards the number of corners.

Options

• corners (optional) Number of corners in the polygon. Default 3.
• long_lim (optional) Array of longitude bounds. Default [-180, 180].
• lat_lim (optional) Array of latitude bounds. Default [-90, 90].

Example

{"area": {"$polygon": {"corners": 5}}}

Returns a GeoJSON polygon with 5 corners, e.g. {"area":{"type":"Polygon","coordinates":[[[-75.26507,81.14973],[-12.29368,64.22995],[60.43231,-15.97496],[-133.6566,-40.40259],[-130.31348,-87.36982],[-75.26507,81.14973]]]}}.

$regex

Returns a RegExp object.

Options

• string (optional) The regular expression string. Default '.*'.
• flags (optional) Flags for the RegExp object. Default ''.

Example

{"expr": {"$regex": {"string": "^ab+c$", "flags": "i"}}}

Returns {"expr":{"$regex":"^ab+c$","$options":"i"}}.

$resolve

Returns a resolved variable from the template context.

Example

For a context of {"myContextVariable":"aContextValue"}}

{"value": {"$resolve":{"variable":"myContextVariable"}}}

Returns {"value": "aContextValue"}.

$substitute

Aliases

• $sub

Substitutes variables in the form of __myVariable__ into the supplied expression and resolves it. The overrides object properties are used before context object properties.

Example

For a context of {"myOverrideVariable":"Tom"}}

{"value": {"$substitute": {"overrides": {"myOverrideVariable": "Bob"}, "expression": "{{faker.name.fullName({ firstName: \"__myOverrideVariable__\" })}}"}}}

Returns {"value": "Bob Smith"}.

Example

For a context of {"mySurnameVar":"Waites"}}

{"value": {"$substitute": {"overrides": {"myFirstNameVar": "Tom"}, "expression": "__myFirstNameVar__ __mySurnameVar__"}}}

Returns {"value": "Tom Waites"}.

Example

For a context of {"contextVar":"Morgan"}}

{"value": {"$substitute": {"expression": "{{faker.name.fullName({ firstName: \"__contextVar__\" })}}"}}}

Returns {"value": "Morgan Jones"}.

Example

{"ageInYears": {"$substitute": {"overrides": {"myDobValue": "1990-11-21"}, "expression": {"$age":{"dob":"__myDobValue__"}}}}}

Returns {"ageInYears": 31}.

The current iteration count of the generator is available in the context as the variable i (zero-indexed). You can use access this using the substitution operator.

Example

{"value": {"$pick": {"array": ["green", "red", "blue"], "element": {"$substitute": {"expression": "__i__"}}}}}

Returns :

{"value": "green"} in the 1st generated JSON,

{"value": "red"} in the 2nd generated JSON,

{"value": "blue"} in the 3rd generated JSON.

$timestamp

Returns a MongoDB Timestamp object.

Options

• t (optional) Set the low value to the specified value. Default random.
• i (optional) Set the high value to the specified value. Default random.

Example

{"ts": {"$timestamp": {"t": 10, "i": 20}}}

Returns {"ts":{"$timestamp":{"t":10,"i":20}}}.

Chance.js

All other $-prefixed strings that don't match any of the built-in operators above are passed on to the Chance.js library. Use the string format for default options, or pass in custom options with the object format.

Some Examples:

{"ip_address": "$ip"}
{"percent": {"$floating": {"min": 0, "max": 100, "fixed": 8}}}
{"birthday": {"$birthday": {"type": "child"}}}
{"phone_no": "$phone"}
{"full_name": {"$name": {"gender": "female"}}}

Advanced Templates

Handlebar rendering

You can use handlebar template strings to build even more complex values, e.g.

mgeneratejs '{"recipient": "{{chance.name()}} <{{chance.email()}}>"}' -n 3

{"recipient":"Lora Jimenez <[email protected]>"}
{"recipient":"Elnora Brewer <[email protected]>"}
{"recipient":"Howard Bryan <[email protected]>"}

mgeneratejs '{"name": "{{faker.name.firstName()}}"}' -n 3

{"name":"Damaris"}
{"name":"Alexzander"}
{"name":"Tiara"}

You can also execute javascript by putting it between these braces:

mgeneratejs '{"value": "{{1+2+3}}"}'

{"value":"6"}

Referencing generated values

Use the -c flag to specify a map of variables for the template context, which will enable to reference the same generated value multiple times in the same template.
These will each be resolved to a variable that can then be referenced in the template JSON as needed, e.g.

mgeneratejs '{"name": {"$resolve":{"variable":"myVariable"}}, "nameAgain": {"$resolve":{"variable":"myVariable"}}}' -c '{"myVariable": "{{faker.name.firstName()}}"}' -n 3

{"name":"Thelma","nameAgain":"Thelma"}
{"name":"Christ","nameAgain":"Christ"}
{"name":"Kaitlin","nameAgain":"Kaitlin"}

You can chain expressions in the context and they will be resolved in the same way as the template (illustrated by this complicated way of setting the context as {"contextVariable": "{{faker.name.fullName({ firstName: 'Tom' })}}"})

mgeneratejs '{"customerName": {"$resolve":{"variable":"contextVariable"}}}' -c '{"contextVariable": {"$substitute": {"overrides": {"myFirstNameVar": "Tom"}, "expression": "{{faker.name.fullName({ firstName: \"__myFirstNameVar__\" })}}"}}}'  -n 1

{"customerName":"Tom Becker MD"}
{"customerName":"Tom Daugherty"}
{"customerName":"Tom Mosciski"}

Difference to mtools' mgenerate script

This is a JavaScript port from the mgenerate script in the mtools library. It is mostly backwards compatible except for the following breaking changes:

1. The "array" operator format is no longer supported, as it was confusing which arguments need to be provided in which order. Instead, use the "object" format with named options. See array shortcut syntax.
2. The "$concat" operator has been renamed to "$join", as this operation is called "join" in many languages, e.g. Python and JavaScript. "$concat" is reserved for a future operator to concatenate arrays.
3. mgeneratejs does not insert documents directly into MongoDB, it only outputs to stdout. It doesn't make sense to re-implement all the authentication options separately, when the resulting objects can simply be piped to mongoimport.

In addition, many more operators are supported through the inclusion of the Chance.js library, and the extended template syntax with handlebar templates.

License

Apache 2.0

With thanks to the original creator, rueckstiess.