csvjson-jsoncsv-cg-lib

1. Introduction

This code has the objective of performing transformations from CSV files to JSON files and vice versa.

This library could be used as part of a component based on Open Integration Hub (OIH) framework or any Nodejs development that requires convertion between this two formats.

The library can be installed from npm page with the next:

npm install csvjson-jsoncsv-cg-lib, npm i csvjson-jsoncsv-cg-lib or yarn install csvjson-jsoncsv-cg-lib

Within "csvjson-jsoncsv-cg-lib" two additional libraries are used: json-2-csv: This library is used for the two transformations, JSON to CSV and CSV to JSON. utils-nxg-cg: is a N3xGen library that is used to print the necessary logs and help with different functions to validate the payload.

As final result in the case of request the "csv_json" transformation the result is going to put the data in JSON format. A resultant sample is shown as follows:

[
    {
        "Name": "John",
        "Last name": "Doe",
        "Address": "120 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "8075"
    },
    {
        "Name": "Philip",
        "Last name": "Miller",
        "Address": "170 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "9065"
    },
    {
        "Name": "Pablo",
        "Last name": "Perez",
        "Address": "444 pedernal",
        "City": "Gomez",
        "Ext": "NJ",
        "CP": "35020"
    }
]

It is important to mention that before the method returns the information, it will be validated that the response is a valid JSON or valid JSON Array. If this is correct, the response will be returned, otherwise an exception will be thrown and the flow or process will stop.

In the case of request the "json_csv" transformation the result is going to put the data in CSV format, and then the resultant data can be saved as file with the ".csv" extension and the custom delimiter. A sample of the result data should look like as follows:

Name,Last name,Address,City,Ext,CP
John,Doe,120 jefferson st.,Riverside,NJ,8075
Philip,Miller,170 jefferson st.,Riverside,NJ,9065
Alejandro,Martinez,623 pedernal,Gomez,NJ,35020

Before the transformation is executed, it is validated that the JSON to be processed is a valid JSON, otherwise an exception will be thrown and the flow or process will be stopped.

2. Methods explanation

2.1. transformCsvToJson

This method is used to convert a file with a CSV extension to a JSON file for data handling preferences. By default, the CSV file has comma as the primary delimiter, and if defined by the user a different separator can be used.

2.2. transformJsonToCSV

This method is used to transform a JSON file or text to a CSV file for data handling preference. Currently, the JSON format has to be a single level since if there were more levels the data could not be transformed.

3. Argument and result explanation

• Args:

  • In the transformJson method we have only a few parameters that are necessary to do the transformation, and they are described below:

    • headers(required): This parameter is to determine if the first line of the CSV file will be taken as headers for the JSON property names or the JSON property names will be created dynamically.
    • content(required): This parameter is the content that will be transformed and has to be received as base64 encoding.

  • In the transformCsv method we have only a one parameter that is necessary to do the transformation, and it is described below:

    • content(required): This parameter is the content that will be transformed and has to be sent encoded in base64.

  Note: Optionaly, for both transformations it can defined a custom delimiter to the CSV files. The default delimiter is , character, but can be configured these other delimiters as required, one at time: & . - | $ * # (ampersand, dot, hyphen, pipe, dollar sign, asterisk and number sign). Also, if an incorrect delimiter is sent, an exception will be thrown and the flow or process will stop.

• Result: The final result varies depending on the process, if it is "csv_json" it will bring a JSON as a result , if it is "json_csv", we will get a data in CSV format: This result is for the "csv_json" process.

[
    {
        "Name": "John",
        "Last name": "Doe",
        "Address": "120 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "8075"
    },
    {
        "Name": "Philip",
        "Last name": "Miller",
        "Address": "170 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "9065"
    },
    {
        "Name": "Pablo",
        "Last name": "Perez",
        "Address": "555 pedernal",
        "City": "Gomez",
        "Ext": "NJ",
        "CP": "35020"
    }
]

In the case of "json_csv", the data is transformed in CSV format ready to be saved with .csv extension:

Name,Last name,Address,City,Ext,CP
John,Doe,120 jefferson st.,Riverside,NJ,8075
Philip,Miller,170 jefferson st.,Riverside,NJ,9065
Alejandro,Martinez,623 pedernal,Gomez,NJ,35020

3. Examples

This part explains some examples of how to send the input parameters to the library and what the result would be. The next represents a valid object with full parameters that can be the input to the library in order to ransform a CSV content (in base64 format):

{
    "headers": true,
    "content":"TmFtZSxMYXN0IG5hbWUsQWRkcmVzcyxDaXR5LEV4dCxDUA0KSm9obixEb2UsMTIwIGplZmZlcnNvbiBzdC4sUml2ZXJzaWRlLE5KLDgwNzUNClBoaWxpcCxNaWxsZXIsMTcwIGplZmZlcnNvbiBzdC4sUml2ZXJzaWRlLE5KLDkwNjUNCkFsZWphbmRybyxNYXJ0aW5leiw2MjMgcGVkZXJuYWwsR29tZXosTkosMzUwMjA="
}

The result for the sample above is the next JSON:

[
    {
        "Name": "John",
        "Last name": "Doe",
        "Address": "120 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "8075"
    },
    {
        "Name": "Philip",
        "Last name": "Miller",
        "Address": "170 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "9065"
    },
    {
        "Name": "Pablo",
        "Last name": "Perez",
        "Address": "666 pedernal",
        "City": "Gomez",
        "Ext": "NJ",
        "CP": "35020"
    }
]

The next object contains a valid one CSV with the parameters to be processed by the library to generate the JSON format. This CSV file contains the # character as a delimiter:

{
    "delimiter": "#",
    "headers": true,
    "content":"TmFtZSNMYXN0IG5hbWUjQWRkcmVzcyNDaXR5I0V4dCNDUA0KSm9obiNEb2UjMTIwIGplZmZlcnNvbiBzdC4jUml2ZXJzaWRlI05KIzgwNzUNClBoaWxpcCNNaWxsZXIjMTcwIGplZmZlcnNvbiBzdC4jUml2ZXJzaWRlI05KIzkwNjUNCkFsZWphbmRybyNNYXJ0aW5leiM2MjMgcGVkZXJuYWwjR29tZXojTkojMzUwMjA="
}

In this sample the CSV content is in base64 encoding. The expected JSON result should be as follows:

[
    {
        "Name": "John",
        "Last name": "Doe",
        "Address": "120 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "8075"
    },
    {
        "Name": "Philip",
        "Last name": "Miller",
        "Address": "170 jefferson st.",
        "City": "Riverside",
        "Ext": "NJ",
        "CP": "9065"
    },
    {
        "Name": "Pablo",
        "Last name": "Perez",
        "Address": "666 pedernal",
        "City": "Gomez",
        "Ext": "NJ",
        "CP": "35020"
    }
]

Another sample is to request the JSON to CSV transformation that takes the content of the JSON and put it into the CSV format. For the sample the JSON data is in base64 encoding as follows:

{
    "content":"WwogICAgewogICAgICAgICJOYW1lIjogIkpvaG4iLAogICAgICAgICJMYXN0IG5hbWUiOiAiRG9lIiwKICAgICAgICAiQWRkcmVzcyI6ICIxMjAgamVmZmVyc29uIHN0LiIsCiAgICAgICAgIkNpdHkiOiAiUml2ZXJzaWRlIiwKICAgICAgICAiRXh0IjogIk5KIiwKICAgICAgICAiQ1AiOiAiODA3NSIKICAgIH0sCiAgICB7CiAgICAgICAgIk5hbWUiOiAiUGhpbGlwIiwKICAgICAgICAiTGFzdCBuYW1lIjogIk1pbGxlciIsCiAgICAgICAgIkFkZHJlc3MiOiAiMTcwIGplZmZlcnNvbiBzdC4iLAogICAgICAgICJDaXR5IjogIlJpdmVyc2lkZSIsCiAgICAgICAgIkV4dCI6ICJOSiIsCiAgICAgICAgIkNQIjogIjkwNjUiCiAgICB9LAogICAgewogICAgICAgICJOYW1lIjogIkFsZWphbmRybyIsCiAgICAgICAgIkxhc3QgbmFtZSI6ICJNYXJ0aW5leiIsCiAgICAgICAgIkFkZHJlc3MiOiAiNjIzIHBlZGVybmFsIiwKICAgICAgICAiQ2l0eSI6ICJHb21leiIsCiAgICAgICAgIkV4dCI6ICJOSiIsCiAgICAgICAgIkNQIjogIjM1MDIwIgogICAgfQpd"
}

The result to this request should be something like this:

Name,Last name,Address,City,Ext,CP
John,Doe,120 jefferson st.,Riverside,NJ,8075
Philip,Miller,170 jefferson st.,Riverside,NJ,9065
Alejandro,Martinez,623 pedernal,Gomez,NJ,35020

In this other sample transformation request is to obtain a CSV for a JSON file. In this case the resultant CSV must use the # character as a delimiter:

{
  "delimiter": "#",
  "content":[
    {
      "Name": "John",
      "Last name": "Doe",
      "Address": "120 jefferson st.",
      "City": "Riverside",
      "Ext": "NJ",
      "CP": "8075"
    },
    {
      "Name": "Philip",
      "Last name": "Miller",
      "Address": "170 jefferson st.",
      "City": "Riverside",
      "Ext": "NJ",
      "CP": "9065"
    },
    {
      "Name": "Pablo",
      "Last name": "Perez",
      "Address": "666 pedernal",
      "City": "Gomez",
      "Ext": "NJ",
      "CP": "35020"
    }
  ]
}

The result is a CSV with # as a separator

Name#Last name#Address#City#Ext#CP
John#Doe#120 jefferson st.#Riverside#NJ#8075
Philip#Miller#170 jefferson st.#Riverside#NJ#9065
Alejandro#Martinez#623 pedernal#Gomez#NJ#35020

To transform from JSON to CSV, consider that if you CSV file doesn't have a headers the property headers is not requires and the result will be a JSON with the following structure, where the name of the elements is "element" + a consecutive number:

[
    {
        "element 1": "John",
        "element 2": "Doe",
        "element 3": "120 jefferson st.",
        "element 4": "Riverside",
        "element 5": "NJ",
        "element 6": "8075"
    },
    {
        "element 1": "Philip",
        "element 2": "Miller",
        "element 3": "170 jefferson st.",
        "element 4": "Riverside",
        "element 5": "NJ",
        "element 6": "9065"
    },
    {
        "element 1": "Alejandro",
        "element 2": "Martinez",
        "element 3": "623 pedernal",
        "element 4": "Gomez",
        "element 5": "NJ",
        "element 6": "35020"
    }
]

If you CSV file has strings that contains separator on it, the string must be wrapped with double quotes "", in the following example the . character is used as a separator and the CSV file should look like as follows:

Name.Last name.Address.City.Ext.CP
John.Doe."120 jefferson st."."Riverside".NJ.8075
Philip.Miller."170 jefferson st."."Riverside".NJ.9065
Alejandro.Martinez."623 pedernal"."Gomez".NJ.35020