DataGen

DataGen is a multi-process test data files generator.

This is handy when you want to generate large test data files (e.g. XMLs, JSONs, CSVs, etc), over multiple processes, utilising available CPU cores on your machine. It's also very easy to generate random numbers, dates, and strings as test data. You only need to create template files, no scripting involved.

Installation

npm install -g datagen 

Usage

Create example header, segment, and footer template files:

datagen init

Generate 8 data files containing 1 million segments in each file, written to data1 ... data8 output files, running over 8 processes, each process is used to generate 1 file:

datagen gen -s 1000000 -w 8 -o data

Limit the number of worker processes that can run concurrently to just 3:

datagen gen -s 1000000 -w 8 -m 3

Templates

DataGen uses three template files: header, segment, and footer. These templates are simple text files which are used to construct a data file in this format:

header
segment 1
segment 2
...
segment N (number of segments)
footer

Example header:

<?xml version="1.0" encoding="UTF-8"?>
<data>

Example segment:

<segment>
  <id>{gen_id}-{worker_id}-{segment_id}</id>
  <name>{first_name()} {last_name()}</name>
  <dob>{date('dd-mm-yyyy')}</dob>
</segment>

Example footer:

</data>

The above templates will generate an XML like this:

<?xml version="1.0" encoding="UTF-8"?>
<data>
<segment>
  <id>1-1-1</id>
  <name>Niels Bryant</name>
  <dob>12-08-1992</dob>
</segment>
<segment>
  <id>1-1-2</id>
  <name>John Bohr</name>
  <dob>01-11-1970</dob>
</segment>
...
</data>

Curly brackets in templates (e.g. JSON format) need to be escaped with double brackets:

{{ "id": "{gen_id}-{worker_id}-{segment_id}", "name": "{first_name()} {last_name()}", "dob": "{date('dd-mm-yyyy')}" }}

Templates can contain the following parameters:

Screenshots

Colophon

Developer's Guide

Build reports:

• Code complexity report
• Unit tests report
• Test coverage report
• Integration tests report
• API Documentation

Articles:

• DataGen Workers Optimisation
• DataGen: Generate Large Test Data Files – Like A Boss