data-restructor-js

When parsing JSON on client-side, the structure of it attracts most of our attention.
If the structure evolves over time, it leads to recurring changes in the code that depends on it.

Features

• Adapter that takes e.g. parsed JSON and transforms it into a uniform structure
• Multiple transformation steps including flattening, removing duplicates, grouping, ...
• Takes descriptions that reflect the incoming structure and define the uniform output
• Reusable and flexible
• Supports most browser including IE 5

Not intended to be used when

• a "backend for frontend" exists, that is responsible for delivering the structure and content the way the client needs it.
• the structure of the data is already stable, well abstracted and/or rather generic.
• the code, that depends on the structure of the data, can easily be changed (only a view lines, same team, ...).

Quickstart

Use the following command to install the library using npm:

npm install data-restructor

Alternatively, the sources can be found inside the source folder:

• datarestructor.js
• templateResolver.js
• describedfield.js

The development artifacts (not minified) can be found inside the devdist folder:

• datarestructor-ie-global-all.js all sources including polyfills merged to be used without module system
• datarestructor-global-all.js all sources without polyfills merged to be used without module system
• datarestructor.js
• templateResolver.js
• describedfield.js

Here are some code examples on how these modules can be imported:

var template_resolver = template_resolver || require("data-restructor/devdist/templateResolver"); // supports vanilla js
var described_field = described_field || require("data-restructor/devdist/describedfield"); // supports vanilla js
var datarestructor = datarestructor || require("data-restructor/devdist/datarestructor"); // supports vanilla js

The built (minified) versions can be found inside the distribution folder:

• datarestructor-ie-global-all-min.js all sources including polyfills merged and minified to be used without module system
• datarestructor-global-all-min.js all sources without polyfills merged and minified to be used without module system
• datarestructor.js
• datarestructor-ie.js (full compatibility with IE)
• templateResolver.js
• templateResolver-ie.js (full compatibility with IE)
• describedfield.js
• describedfield-ie.js (full compatibility with IE)

Code Documentation

The code documentation is generated using JSDoc and is published using GitHub Pages at https://joht.github.io/data-restructor-js.

Build all

Use the following commands to build and package the module. A list of all commands can be found in COMMANDS.md.

npm install merger-js -g
npm install
npm run package

Note: merger.js prompts to select a source file. Please select "ALL" using the arrow keys and press enter to continue.

Example

As a starting point you may have a look at the following example.
A running, comprehensive example can be found here: DataRestructorUseCaseTest.js

Input Object

{
    "responses": [
        {
            "hits": {
                "total": {
                    "value": 1
                },
                "hits": [
                    {
                        "_source": {
                            "iban": "AT424321012345678901",
                            "accountnumber": "12345678901",
                            "customernumber": "00001234567",
                            "currency": "USD",
                            "tags": [
                                "active",
                                "online"
                            ]
                        }
                    }
                ]
            }
        }
    ]
}

Code

function restructureJson(jsonData) {
  var allDescriptions = [];
  allDescriptions.push(summariesDescription());
  allDescriptions.push(detailsDescription());
  return new datarestructor.Transform(allDescriptions).processJson(jsonData);
}

function summariesDescription() {
  return new datarestructor.PropertyStructureDescriptionBuilder()
    .type("summary")
    .category("account")
    .propertyPatternEqualMode()
    .propertyPattern("responses.hits.hits._source.accountnumber")
    .groupName("summaries")
    .groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
    .build();
}

function detailsDescription() {
  return new datarestructor.PropertyStructureDescriptionBuilder()
    .type("detail")
    .category("account")
    .propertyPatternTemplateMode()
    .propertyPattern("responses.hits.hits._source.{{fieldName}}")
    .groupName("details")
    .groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
    .groupDestinationPattern("account--summary--{{index[0]}}--{{index[1]}}")
    .build();
  }

Output Java Object

An Javascript object with mainly this structure (see DescribedEntry for more details) and content is returned, when the function restructureJson from above is called:

category: "account"
displayName: "Accountnumber"
fieldName: "accountnumber"
type: "summary"
value: "12345678901"
details:
  - category: "account"
    type: "detail"  
    displayName: "Iban"
    fieldName: "iban"
    value: "AT424321012345678901"
  - category: "account"
    type: "detail"
    displayName: "Accountnumber"
    fieldName: "accountnumber"
    value: "12345678901"
  - category: "Konto"
    type: "detail"
    displayName: "Customernumber"
    fieldName: "customernumber"
    value: "00001234567"
  - category: "Konto"
    type: "detail"
    displayName: "Currency"
    fieldName: "currency"
    value: "USD"
  - category: "Konto"
    type: "detail"
    displayName: "Tags"
    fieldName: "tags"
    value: "active"
  - category: "Konto"
    type: "detail"
    displayName: "Tags"
    fieldName: "tags"
    value: "online"
  - category: "Konto"
    type: "detail"
    displayName: "Tags"
    fieldName: "tags_comma_separated_values"
    value: "active, online"

Transformation Steps

1. Flatten hierarchical data object

The input data object, e.g. parsed from JSON, is converted to an array of point separated property names and their values. For example this structure...

{
    "responses": [
        {
            "hits": {
                "total": {
                    "value": 1
                },
                "hits": [
                    {
                        "_source": {
                            "accountnumber": "123"
                        }
                    }
                ]
            }
        }
    ]
}

...is flattened to...

responses[0].hits.total.value=1
responses[0].hits.hits[0]._source.accountnumber=123

2. Add array value properties ending with "_comma_separated_values"

To make it easier to e.g. display array values like tags, an additional property is added that combines the array values to a single property, that contains the values in a comma separated way. This newly created property gets the name of the array property followed by "_comma_separated_values" and is inserted right after the single array values.

For example these lines...

responses[0].hits.total.value=1
responses[0].hits.hits[0]._source.tags[0]=active
responses[0].hits.hits[0]._source.tags[1]=online

...will lead to an additional property that looks like this...

responses[0].hits.hits[0]._source.tags_comma_separated_values=active, online

3. Attach description to matching properties

For every given description, all properties are searched for matches. If a description matches a property, the description gets attached to it. This can be used to categorize and filter properties. The description builder accepts these ways to configure property matching:

• Equal Mode (default):
  The property name needs to match the described pattern exactly. It is not needed to set equal mode. The field name will be (by default) taken from the right most (after the last separator .) element of the property name. In the example below the field name will be "accountnumber". Example:

  new datarestructor.PropertyStructureDescriptionBuilder()
  .propertyPatternEqualMode()
  .propertyPattern("responses.hits.hits._source.accountnumber")
  ...

• Pattern Mode:
  The property name needs to start with the described pattern. The pattern may contain variables inside double curly brackets.
  The variable {{fieldName}} is a special case which describes from where the field name should be taken. If {{fieldName}} is not specified, the field name will be taken from the right most (after the last separator .) element of the property name, which is the same behavior as in "Equal Mode". This mode needs to set using propertyPatternTemplateMode, since the default mode is propertyPatternEqualMode. Example:

  new datarestructor.PropertyStructureDescriptionBuilder()
  .propertyPatternTemplateMode()
  .propertyPattern("responses.hits.hits._source.{{fieldName}}")
  ...

• Index Matching (Optional):
  If the source data is structured in an top level array and all property names look pretty much the same it may be needed to describe data based on the array index. The index of an property is taken out of its array qualifiers.
  For example, the property name responses[0].hits.hits[1]._source.tags[2] has the index 0.1.2.
  Index Matching can be combined with property name matching. This example restricts the description to the first top level array:

  new datarestructor.PropertyStructureDescriptionBuilder()
  .indexStartsWith("0.")
  ...

4. Removing duplicates (deduplication)

To remove duplicate properties or to override properties with other ones when they exist, a deduplicationPattern can be defined. Variables (listed below) are put into double curly brackets and will be replaced with the contents of the description and the matching property.
If there are two entries with the same resolved deduplicationPattern (=_identifier.deduplicationId), the second one will override the first (the first one will be removed). Example:

new datarestructor.PropertyStructureDescriptionBuilder()
.deduplicationPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}--{{fieldName}}")
...

5. Grouping

Since data had been flattened in the step 1., it is structured as a list of property names and their values. This non-hierarchical structure is ideal to add further properties, attach descriptions and remove duplicates. After all, a fully flat structure might not be suitable to display overviews/details or to collect options. The groupName defines the name of the group attribute (defaults to "group" if not set). The groupPattern describes, which properties belong to the same group.
Variables (listed below) are put into double curly brackets and will be replaced with the contents of the description and the matching property.
The groupPattern will be resolved to the _identifier.groupId. Every property, that leads to a new groupId gets a new attribute named by the groupName, where this entry and all others of the same group will be put into. Example:

new datarestructor.PropertyStructureDescriptionBuilder()
.groupName("details")
.groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
...

6. Moving groups (destination group)

After grouping in step 5., every property containing a group and the remaining non-grouped properties are listed one after another. To organize them further, a group can be moved beneath another (destination) group. The groupDestinationPattern contains the pattern of the group to where the own group should be moved. Variables (listed below) are put into double curly brackets and will be replaced with the contents of the description and the matching property.
Optionally, the groupDestinationName can be specified to rename the group when it is moved. Default is the value of groupName. Example, where the details group is moved to the summary, because the group destination pattern of the details resolves to the same id as the resolved group pattern of the summary:

var summaryDescription = new datarestructor.PropertyStructureDescriptionBuilder()
 .category("account")
 .type("summary")
 .groupName("summaries")
 .groupPattern("{{category}}--{{type}}--{{index[0]}}--{{index[1]}}")
 ...

var detailsDescription = new datarestructor.PropertyStructureDescriptionBuilder()
.groupDestinationPattern("account--summary--{{index[0]}}--{{index[1]}}")
.groupDestinationName("details")
...

7. Convert data into an array of DescribedFields

The result is finally converted into an array of DescribedDataFields.

Types, fields, variables

This section lists the types and their fields in detail (mostly taken from jsdoc). Every field can be used as variable in double curly brackets inside pattern properties. Additionally, single elements of the index can be used by specifying the index position e.g. {{index[0]}} (first), {{index[1]}} (second),...

PropertyStructureDescription (input description)

• type - ""(default). Some examples: "summary" for e.g. a list overview. "detail" e.g. when a summary is selected. "filter" e.g. for field/value pair results that can be selected as search parameters.
• category - name of the category. Default = "". Could contain a symbol character or a short domain name. (e.g. "city")
• abbreviation - ""(default). One optional character, a symbol character or a short abbreviation of the category.
• image - ""(default). One optional path to an image resource.
• propertyPatternTemplateMode - boolean "false"(default): property name needs to be equal to the pattern. "true" allows variables like {{fieldname}} inside the pattern.
• propertyPattern - property name pattern (without array indices) to match. A pattern may contain variables in double curly brackets {{variable}}. See also: variables, further details
• indexStartsWith - ""(default) matches all ids. String that needs to match the beginning of the id. E.g. "1." will match id="1.3.4" but not "0.1.2".
• groupName - name of the property, that contains grouped entries. Default="group".
• groupPattern - Pattern that describes how to group entries. "groupName" defines the name of this group. A pattern may contain variables in double curly brackets {{variable}}. See also: variables, further details
• groupDestinationPattern - Pattern that describes where the group should be moved to. Default=""=Group will not be moved. A pattern may contain variables in double curly brackets {{variable}}. See also: variables, further details
• groupDestinationName - (default=groupName) Name of the group when it had been moved to the destination.
• deduplicationPattern - Pattern to use to remove duplicate entries. A pattern may contain variables in double curly brackets {{variable}}. See also: variables, further details

DescribedDataField

This is the data structure of a single output element representing a field. Beside the properties described below, the described data field can also contain custom properties containing groups (arrays) of sub fields of type DescribedDataField.

Before version 3.0.0 this structure was named DescribedEntry and also contained internal fields.
Since 3.0.0 and above, DescribedEntry is only used internally and is not public any more.

DescribedDataField Public Fields

• category - category of the result from the PropertyStructureDescription using a short name or e.g. a symbol character
• type - type of the result from PropertyStructureDescription
• abbreviation - one optional character, a symbol character or a short abbreviation of the category
• image - one optional path to an image resource
• index - contains an array of numbers representing the hierarchical index for list entries (and their sub lists ...). Example: "responses[2].hits.hits[4]._source.name" will have an index of [2,4].
• groupNames - contains an array of String names. Every name represents a group that had been dynamically added as property. Groups should be added using DescribedDataFieldGroup, which will also update the group names.
• displayName - display name extracted from the point separated hierarchical property name, e.g. "Name"
• fieldName - field name extracted from the point separated hierarchical property name, e.g. "name"
• value - content of the field

DescribedDataField Public Functions

Since version 3.0.0 and above, there are no functions any more.

Described groups

• "name of described group" as described in PropertyStructureDescription
• "names of moved groups" as described in PropertyStructureDescription of the group that had been moved

DescribedDataFieldGroup

This helper was added with version 3.0.0. It adds groups to DescribedDataFields. These groups are dynamically added properties that contain an array of sub fields also of type DescribedDataField.

DescribedDataFieldGroup Public Functions

• addGroupEntry(groupName, entry) Adds an entry to the given group. If the group does not exist, it will be created and added to the "groupNames".
• addGroupEntries(groupName, entries) Adds an array of entries to the given group. If the group does not exist, it will be created and added to the "groupNames".

DescribedEntry

Since 3.0.0 and above, DescribedEntry is only used internally and is not public any more. It is documented here for sake of completeness and for maintenance purposes. See JSDoc for a more comprehensive reference.

Properties

• describedField - contains the DescribedDataField
• isMatchingIndex - true, if _identifier.index matches the described "indexStartsWith"
• _identifier - internal structure for identifier. Avoid using it outside since it may change.
• _identifier.index - array indices in hierarchical order separated by points, e.g. "0.0"
• _identifier.value - the (single) value of the "flattened" property, e.g. "Smith"
• _identifier.propertyNamesWithArrayIndices - the "original" flattened property name in hierarchical order separated by points, e.g. "responses[0].hits.hits[0]._source.name"
• _identifier.propertyNameWithoutArrayIndices - same as propertyNamesWithArrayIndices but without array indices, e.g. "responses.hits.hits._source.name"
• _identifier.groupId - Contains the resolved groupPattern from the PropertyStructureDescription. Entries with the same id will be grouped into the "groupName" of the PropertyStructureDescription.
• _identifier.groupDestinationId - Contains the resolved groupDestinationPattern from the PropertyStructureDescription. Entries with this id will be moved to the given destination group.
• _identifier.deduplicationId - Contains the resolved deduplicationPattern from the PropertyStructureDescription. Entries with the same id will be considered to be a duplicate and hence removed.
• _description - PropertyStructureDescription for internal use. Avoid using it outside since it may change.

Template Resolver

An simple template resolver is included and provided as separate module. Here is an example on how to use it:

var template_resolver = require("templateResolver");
var sourceDataObject = {type: "MyType", category: "MyCategory"};
var resolver = new template_resolver.Resolver(sourceDataObject);
var template = "{{type}}-{{category}}";
var resolvedString = resolver.resolveTemplate(template);
//resolvedString will contain "MyType-MyCategory"

Template Resolver Public Functions

• resolveTemplate - resolves the given template string. The template may contain variables in double curly brackets:
  • All public fields can be used as variables, e.g. "{{fieldName}}", "{{displayName}}", "{{value}}".
  • Described groups that contain an array of described entries can also be used, e.g. "{{summaries[0].value}}".
  • Parts of the index can be inserted by using e.g. "{{index[1]}}".
  • Besides the meta data, a described field can be used directly by its "fieldName", e.g. "{{customernumber}}" will be replaced by 123, if the structure contains fieldname="customernumber", value="123". This also applies to sub groups, e.g. "{{details.customernumber}}" will be replaced by 321, if the structure contains details[4].fieldname="customernumber", details[4].value="321".

TransformConfig

An comprehensive and up to date reference can be found here: TransformConfig JSDoc.

The restructured data is by nature hierarchical and may contain cyclic data references. Fields may contain groups of fields that may contain groups of fields.... Since JSON can't be generated out of objects with cyclic references, sub-structures are expressed by copies. That leads to recursion and duplication, that need to be limited. This can be configured here.

TransformConfig Properties

• debugMode boolean value, that enables/disables detailed logging
• maxRecursionDepth numeric value that defines the maximum recursion depth
• removeDuplicationAboveRecursionDepth numeric value that defines the recursion depth, above which duplications inside groups will be removed.

Public functions (provides by "Transform")

• enableDebugMode(boolean) boolean value, that enables/disables detailed logging
• setMaxRecursionDepth(number) numeric value that defines the maximum recursion depth
• setRemoveDuplicationAboveRecursionDepth(number) numeric value that defines the recursion depth, above which duplications inside groups will be removed.

Related blog articles

• Most effective ways to push within GitHub Actions
• Continuous Integration for JavaScript with npm

References

• Mozilla MDN web docs - polyfill for 'Array.filter' for browser compatibility
• Mozilla MDN web docs - polyfill for 'Array.forEach' for browser compatibility (references es5.github.io)
• Mozilla MDN web docs - polyfill for 'Array.indexOf' for browser compatibility
• Mozilla MDN web docs - polyfill for 'Array.isArray' for browser compatibility
• Mozilla MDN web docs - polyfill for 'String.startsWith' for browser compatibility
• Token Posts - polyfill for 'Object.keys' for browser compatibility

Credits

Although this project doesn't use any runtime dependencies, it is created using these great tools:

• ESLint - MIT License
• istanbul-badges-readme - MIT License
• Jasmine - MIT License
• JSDoc - Apache Licence 2.0
• merger-js - GNU General Public License v3.0
• NYC aka Istanbul - ISC License
• PARCEL - MIT License
• merger-js - GNU General Public License v3.0