DTL

WebSite | Repository | Wiki | Bug Reports | Live Help (discord) | By Jay Kuri

DTL, short for Data Transformation Language, is a versatile tool ideal for managing and transforming structured data, especially from sources like NoSQL databases and APIs. It seamlessly integrates into your JavaScript projects, whether you're working in a Node.js environment or directly in the browser. Additionally, DTL offers powerful command-line tools that excel in handling various formats, including JSON, YAML, CSV, and plain text. These tools are perfect for processing and exploring existing data sets.

Whether you need something as simple as JSON templating, more complex tasks like format conversion, or intricate data restructuring, DTL stands out as an invaluable resource for developers dealing with any form of structured data manipulation.

If your work involves manipulating structured data, and what development doesn't, DTL offers an efficient and easy to understand toolset to help you.

What DTL looks like

DTL looks like JSON and has a very familiar syntax:

{
    "out": {
        "name": "(: &( $first_name ' ' $last_name ) :)",
        "order_total": "(: $order_subtotal + $tax_amt :)",
        "origin": "Import from legacy database",
        "address": {
            "street": "(: $address1 :)",
            "unit": "(: $address2 :)",
            "city": "(: $addr_city :)",
            "state": "(: $addr_state :)",
            "postal": "(: $addr_zip :)"
        }
    }
}

We'll explore this example in detail a bit later.

Installing DTL

DTL can be installed within your project, or globally for command-line use.

Global Installation

For global access to DTL's command-line tools, install DTL globally using npm or yarn:

• Using npm

  npm install -g dtl-js

• Using yarn

  yarn global add dtl-js

Local Installation

To install DTL as part of your project, navigate to your project directory and run:

• Using npm

  npm install dtl-js

• Using yarn

  yarn add dtl-js

Verifying the Installation

After installation, you can verify the installation of DTL:

• For global installations, check the version to ensure it's installed correctly:

  dtl --version

• For local installations, you can check your package.json to see if dtl-js is listed under dependencies.

What DTL is for...

DTL, is built for transforming and manipulating data structures. At its most basic level, DTL can act as a templating tool for JSON or YAML, enabling straightforward data reformatting and restructuring. However, DTL's capabilities extend far beyond simple templating. DTL is ideal for complex tasks such as data extraction, conversion, and preparation for analysis or reporting. Its simple syntax and robust functions facilitate the clear and concise expression of complex data transformations, making DTL an extremely useful tool for a wide range of data processing needs.

Basic Structure

In DTL, transforms are generally defined within a JSON object, which forms the transform library. Each key in this JSON object is a distinct transform, functioning like a template or function for how input data should be outputted. The out key is special as it's the default transform used by DTL.apply(), akin to the main function in other languages.

In most cases, the value of out will be an object that looks like your desired output structure, guiding the transformation of input data to this format. This library can have multiple transforms, each capable of referencing and calling one another. When a transform is executed through DTL.apply(), the provided data is represented as $. within the transform.

Syntax Essentials

1. Happy Tags

Every DTL expression must be enclosed in happy tags: (: :). Happy tags tell DTL to do something. Anything not in Happy tags is passed through untouched. Happy tags are called that because they resemble happy emoji.

2. Accessing data

Reference data using dot notation, with the root represented as $. For nested data access, use dot notation, e.g., $user.address.street Data can be referenced with or without a leading ., in other words, $.user and $user are equivalent.

And here's a bit of good news: in DTL, trying to access a property of a non-existent or non-object value won't send you into the dreaded JavaScript spiral of Cannot read properties of undefined. In DTL, it just sensibly returns undefined. No drama, no crash – just the peace of mind knowing DTL has got your back in these scenarios.

3. No Commas

In DTL expressions, commas are treated like whitespace and are not necessary.

4. Helper Functions and Transform Calls

Use built-in helper functions for complex operations, e.g., math.round($number). To call another transform in the library, use ($input_data -> transform_name). In this example, $input_data would be sent to transform_name and the result would be the new data.

Operations look like this: $data op $otherdata, e.g., $number1 + $number2.

5. Parentheses for Order

Use parentheses to control the order that things should be done, e.g., ($number1 + $number2) * $number3. DTL understands mathematical order of operations, but there's no PEMDAS for data types, so parenthesis will help make your intentions clearer.

6. Conditional Syntax

For conditionals, use ?(: $condition $truevalue $falsevalue). You can use this to send data to one or another transform depending on some condition: ?(!empty($address) ($. -> process_address) ($. -> no_address))

In this example, ?(!empty($address) ($. -> process_address) ($. -> no_address)), DTL checks if $address is not empty. If it's not, the data is processed with the process_address transform; if $address is empty, the no_address transform is used instead. This showcases DTL's ability to elegantly handle conditional data routing based on the presence or absence of data.

7. Static and Dynamic Values

A mix of static (literal values) and dynamic (DTL expressions) is common in transforms. Only dynamic expressions use happy tags.

8. Iteration

You can loop or iterate over lists of data. In all helpers that iterate, like map or grep, use $item, $index, and $all for referencing. $item is the current item. $index is the index of the current item and $all is the entire list or object.

9. Scope

Transforms only have access to what was given to them (from DTL.apply() or another transform in the library) and have no way to access information other than that. This is a good thing, it means your transform will always do exactly what you want. Within a transform, $. refers to the input data of that transform.

10. DTL Math is Real Math

In DTL, mathematical operations are carried out using Arbitrary Precision Mathematics. This approach is a game-changer, especially when you compare it to JavaScript, where 0.1 + 0.2 equals 0.30000000000000004 (seriously, try it in your web or Node.js console). In DTL, it simply equals 0.3, just as you would logically expect. This means that when you're doing calculations in DTL, you're getting the exactly correct answer every time. DTL's precision makes it ideal for scenarios where mathematical accuracy isn't just a preference, it's a necessity.

11. The concept of empty

In DTL, the concept of empty is key. Empty in DTL means the data has no meaningful value, e.g. undefined, null, an empty string, array, or object. The empty() function returns true if a value is 'empty'

If you've ever wondered why [] evaluates to true or been bitten by if(formfield) evaluating to false if someone entered 0, we feel your pain. In most cases, you really want to know if a variable has a meaningful value. empty() is here to help.

Complementing this, the fne() (First Non-Empty) function scans a list of values and returns the first one that is not considered empty, streamlining the selection of valid data from multiple sources. It looks like this: fne($first_place_to_look $second_place_to_look defaultvalue)

Example in JSON

Here's an example of DTL:

{
    "out": {
        "name": "(: &( $first_name ' ' $last_name ) :)",
        "order_total": "(: $order_subtotal + $tax_amt :)",
        "origin": "Import from legacy database",
        "address": {
            "street": "(: $address1 :)",
            "unit": "(: $address2 :)",
            "city": "(: $addr_city :)",
            "state": "(: $addr_state :)",
            "postal": "(: $addr_zip :)"
        }
    }
}

In this JSON transformation example:

• The name field uses the &() helper to concatenate $first_name and $last_name with a space in between.
• The order_total field does some simple math, adding $order_subtotal and $tax_amt together.
• origin is a static string, "Import from legacy database".
• The address field is an object containing multiple expressions. This shows how your transforms mirror your desired output format.

### There is no step 3.

Using DTL is as straightforward as it gets: 

1. **Create Your Transform**: Defining how you want your output data to
   look. This step involves crafting the template or transformation logic in
   a JSON object, specifying how the input data should be processed and
   structured.

2. **Call `DTL.apply()`**: Once your transform is ready, the next step is to
   apply it to your data. This is done using the `DTL.apply()` method, which
   takes your input data and the transform, performs the transformation as defined,
   and returns the result.

And that's it — there is no step 3. DTL's design is centered around this
simplicity, making data transformations easy and accessible.

Below is a practical example demonstrating these two steps in a Node.js script:

### Putting It All Together: Example Node.js Script

Now that we've covered the basics of DTL, let's see it in action with a
simple Node.js script. This example demonstrates how to use DTL to
transform some sample data and output the result.

```javascript
// Import the DTL module
const DTL = require('dtl-js');

// Example input data
const inputData = {
    first_name: "John",
    last_name: "Doe",
    order_subtotal: 100,
    tax_amt: 20,
    address1: "123 Main St",
    address2: "Apt 4",
    addr_city: "Springfield",
    addr_state: "IL",
    addr_zip: "62704"
};

// Example DTL transform
const transformLibrary = {
    "out": {
        "name": "(: &( $first_name ' ' $last_name ) :)",
        "order_total": "(: $order_subtotal + $tax_amt :)",
        "origin": "Import from legacy database",
        "address": "(: $. -> formatAddress :)"
    },
    "formatAddress": {
        "street": "(: $address1 :)",
        "unit": "(: $address2 :)",
        "city": "(: $addr_city :)",
        "state": "(: $addr_state :)",
        "postal": "(: $addr_zip :)"
    }
};

// Applying the transform to the input data
const transformedData = DTL.apply(inputData, transformLibrary);

console.log("Transformed Data:", JSON.stringify(transformedData, undefined, 4));

This script demonstrates how to import DTL, define a transform library, apply it to some sample data, and output the transformed result. It's a practical example of how DTL can be utilized in a typical Node.js environment to manipulate and transform data structures.

As you can see, integrating DTL into your project is super simple. With its simple syntax and powerful capabilities, DTL is all set to make your job easier. Now, go use it to streamline your code, make your workflows more efficient or just do some awesome stuff!

Full Editor support

Working with DTL is made significantly easier thanks to comprehensive editor support. Full syntax highlighting in the editor bundles not only clarifies DTL syntax but also enhances your coding efficiency. With snippets for all built-in functions, these bundles facilitate a more intuitive and error-free coding experience. We encourage you to download the appropriate bundle for your editor of choice.

Look for DTL in the VSCode extension marketplace, or if you use Vim, Sublime Text or other Textmate compatible editors, you can download the language support files from the Releases page of the DTL Repository Streamline your coding with these powerful features for an improved, error-free DTL coding experience.

Common Problems That DTL Solves

DTL is adept at addressing a wide range of frequently encountered data transformation challenges. Here are some common problems where DTL can be particularly beneficial:

1. Creating Data Templates: Effortlessly generate structured data from templates. DTL excels in scenarios like auto-generating JSON configurations or preparing data payloads for APIs, making these tasks quick and error-free.

2. Simple Data Transformations: Whether it's renaming keys in JSON objects, restructuring data (such as flattening nested structures), or performing basic aggregations and calculations, DTL simplifies these common tasks.

3. Format Conversion: DTL streamlines the conversion between various data formats. Transforming data from JSON to CSV, XML to JSON, or YAML to JSON becomes a hassle-free process.

4. Data Cleaning and Normalization: Clean or normalize data from diverse sources to fit specific standards. Tasks like removing unwanted fields, standardizing date formats, or converting data types are made straightforward with DTL.

5. Data Validation: Ensure that incoming data conforms to specified criteria or formats. DTL's data validation capabilities are particularly useful for data from external sources or user inputs.

6. Extracting Specific Data from Complex Structures: Extracting necessary information from complex or nested data structures, such as certain fields from deeply nested JSON objects, is made efficient and simple.

7. Automating Repetitive Data Tasks: DTL helps automate repetitive data manipulation tasks, saving time and minimizing the risk of errors in routine operations.

Using DTL, you can address these common data transformation challenges more efficiently, allowing you to focus on the more intricate aspects of your projects.

TL;DR

DTL Repl demo

MOAR Usage

To open the dtlr data exploration environment:

$ dtlr filename.json

To do bulk data manipulation on the command line:

See available command line options:

$ dtl -h

(Movie examples below use sample movie data from Awesome JSON Datasets )

# Convert movies.json to yaml
$ dtl -o movies.yaml movies.json

# How many movies are listed?
$ dtl -e 'length($.)' movies.json`

# Get all the movies where Bill Murray was in the cast
$ dtl -e 'grep($. "(: member($item.cast `Bill Murray`) :)")' movies.json

#  Extract the `name` field in the JSON file in uppercase.
$ dtl -e 'uc($name)' data.json

# Apply the transform expression specified in the `transform.dtl` file to the CSV input file.
$ dtl -f transform.dtl input.csv

# Process `input.json`, output CSV data in `output.csv` including all fields found in input.json.
$ dtl -o output.csv input.json`

# Process `input.json`, output CSV data on stdout using the fields `first_name`, `last_name` and `primary_email` 
dtl -O csv -C first_name,last_name,primary_email input.json

Is DTL safe?

Unlike regular code, the output of DTL can only include the information provided to the DTL call, so DTL transforms are much safer to use than the code that would be required to produce the same output. They're also a heck of a lot easier to read... AND since they are self-contained and don't refer to your code, they are safe and easy to share.

DTL can be used within javascript code (node.js and browser, and even inside MongoDB) or it can be used on the command line with the DTL cli tools.

Why should I care?

DTL is interesting for several reasons:

• Clarity - DTL is purpose-built for data transformation and only data transformation. It is not intended to be a general-purpose programming language and is therefore simple to learn and free of unnecessary components.

• Portable - DTL transforms are self-contained and transferrable between applications. Since they can be stored as JSON, they can even be kept in your database.

• Security - DTL transforms only have access to the data that was provided as input. DTL transforms have no other variable or system access, so they are much safer to use than custom code.

• Stateless - DTL transforms have no access to previous state, only to the data provided and therefore avoid bugs related to bleed over or inadvertant modification, one of the most common sources of bugs.

• Provable - It is trivial to create a DTL transform to verify the output of another. This obviously allows for simple test-creation. What may not be obvious is that these verification transforms can be used to check data at run-time.

• Non-linear - DTL transforms define how to arrive at the desired data. They do not define a sequence of steps. This means that each expression is independent and not subject to bugs due to issues that occurred in other expressions.

• Stable - DTL has been in use in production since 2013 and has been its own separate project since 2015. It is being used in many production applications, handling many millions of transformations every day.

• DTL is a language with an implementation. The DTL npm module is only one implementation of the DTL language. The DTL module contains hundreds of tests that verify the language is behaving properly. This allows DTL to be implemented and verified in any programming language.

Where did DTL come from?

Truth be told, DTL was not originally intended to be it's own thing. DTL began as an expression language inside a meta-programming engine built by Jay Kuri (me) during my work at Ionzero, a company I founded. One of the first applications of this engine was a system built to handle linking other systems together. I created the language out of the need for a way to define how to map data from one system to another without resorting to hard-coded custom code.

I also realized during the course of this work that DTL could be used for far more than I ever had originally envisioned. As a result of this realization, over time, I refined the DTL language and eventually extracted it into a self-contained module that could be used in any system and proceeded to do so.

I decided to release DTL as open source in the hopes that others would find it as useful and as powerful as I have.

The DTL command line tools

As mentioned earlier in this document, if you have installed the DTL package globally with npm or yarn, you will have two command line tools for working with DTL. The dtl cli tool works on bulk data

If you want to just take DTL for a spin without coding you can use the dtlr tool. The dtlr cli tool is an interactive REPL (Read Execute Print Loop) tool you can use to test out expressions and get help.

The dtl cli tool works on bulk data and is designed to process CSV, yaml and JSON, as well as JSONLines data. It can produce CSV, yaml, JSON and JSONLines data as well, regardless of whether the input data was the same type. You can learn more about how to use it by using the dtl -h command. Note that by default it sends its output to stdout. If you'd rather have the output go into a file, use the -o filename.json option.

Feedback and where to get help

We are always looking for constructive feedback. If you have ideas on how we might improve DTL, please reach out. If you are looking for help on how to use DTL, we also want to hear from you.

For help learning DTL, the dtlr tool has help built in by using the .help command. You can also You can visit the docs or look at the DTL Expression Syntax. You can also view all the helper function docs here.

If you want to see examples of DTL or try it out, head to the DTL website and click the 'Try DTL' button to explore DTL right in your browser.

If you want to see more examples, you can take a look at the Test Suite where you can find an example of just about anything DTL can do.

If you have ChatGPT Plus, you can talk to the DTL Helper to get help with DTL.

If you want something a bit more real-time and real-person, you can talk with us on the DTL discord.

And, if you encounter a bug, please don't hesitate to file an Issue.