Structured Data Transfer Format - SDK v1

The Structured Data Transfer Format (sdTF) is an API-neutral exchange and storage format for trees of data items as used by parametric 3D modeling software like Grasshopper®. This open, efficient, and easily extensible data format enables the exchange of complex data structures, independent of software vendors. See the sdTF documentation for more details.

The aim of this SDK is to provide an easy-to-use, lightweight, cross-platform library to interact with sdTF files. The core features consist of an sdTF reader, including a flexible and extensible data parser, and an sdTF writer with a built-in structural optimizer.

This module can be used in Node.js and modern Browsers, and exposes the respective TypeScript declarations.

Installation

npm i @shapediver/sdk.sdtf-v1

Additionally, the following plugins can be included as well to improve the handling of specific data types:

• Primitive type integration - added by default!
• Geometry type integration - added by default!
• Rhino3dm type Integration

However, it is easy to implement new integrations for custom types!
See integrations for more information.

Usage

This code initializes the SDK, reads the sdTF file at the given path and prints the file's JSON content.

// index.ts
import { create, SdtfSdk } from "@shapediver/sdk.sdtf-v1"

(async () => {
    const sdk: SdtfSdk = await create()
    const sdTF = await sdk.createParser().readFromFile("./test_data/sdTF_spec_example.sdtf")
    console.log(sdk.createFormatter().prettifyReadableAsset(sdTF))
})()

SDK initialization and configuration

The SDK handles sdTF content in a structural way. It allows to read and write sdTF structures, and provides data access functions to the user. According to the sdTF v1 specification, data content is stored in data items and attributes, whereby their respective content type is specified via typehint components. However, the SDK is not concerned with data content types. Instead, data content is returned with type unknown and the responsibility of explicitly typing the data content is passed on to the user. As long as the sdTF component structure is valid, sdTFs can be read and written without issues.

To manage the mapping, validation and optimization of content data, the SDK can be extended by various integrations. sdTF integrations provide an easy way to customize reading and writing of data content for specific data types. By default, the SDK instantiates the integrations for primitive types and geometry types. The default integrations can be adopted by specifying a customized list of integrations in the integrations-array of the SdtfConfig object.

By default, the SDK does not use any authorization token for its HTTP calls. However, ShapeDiver models can be configured so that all interactions with them require a JSON Web Token (JWT) for authorization. Thus, a valid JWT must be provided when the SDK is instantiated and the sdTF file is accessed remotely (readFromUrl). Otherwise, a401 HTTP status is returned when the sdTF is fetched.

import { create } from "@shapediver/sdk.sdtf-v1"

create({
    // Remove the default integrations from the sdTF
    integrations: [],
    
    // Add a JWT for HTTP authorization
    authToken: "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL3d3dy5zaGFwZWRpdmVyLmNvbS9hcGkvdjEvdG9rZW5zIiwic3ViIjoiMWM0MzY2MzAtMWYxNy00Y2QyLWFmMTItNDNiYTMzZmVkMDYyIiwiYXVkIjoiOTIwNzk0ZmEtMjQ1YS00ODdkLThhYmUtYWY1NjlhOTdkYTQyIiwiZXhwIjoxNjY3NDc4MDUzLCJzY29wZSI6Imdyb3VwLmV4cG9ydCBncm91cC52aWV3IiwiaWF0IjoxNjY3NDc0NDUzLCJzZF91c2VyX2lkIjoiOTQ4MzJlN2YtMGNjNS00YjE4LWIxNGItOTBkN2FlZTYxYzIzIn0.AKkzs-mCW_3qA3XISMiaW6Bp_MFPVoUnWPWcPllUOZ1Ve5K_tFd0xxpT_T0AoUs8OxQZBXujdKojJLj5sbycKA7X9IEGomQBYjCoTJsQGeafFJW_LSrWb9Z4L9xTu0g02UcvKbwSxyIxLug0pVunklSNN382sgbcKVt6ZapT5a8YGaH5LjYCHVb90OTTQ2JUurtdyqyBLJ1CJbszdkggZuyV2uHhSgJ--jrxcOq_lBgI-tuPj4cbQx340vdhPFXTZA9NAnk6fuI8kfSSej-BQpkdHBi_FAhxlQf5AZov6BonaKl8KIvcFi2Zk77jyB6fjfWwXsT5s0kNVaali58PXQ"
}).then(sdk => {
    // use sdk here
})

Reading a sdTF file

The ISdtfParser object exposes functionality to read an existing sdTF file from various sources and parsing the sdTF structure.

parser.readFromBuffer:
Parses the sdTF file from the given ArrayBuffer and returns a new ISdtfReadableAsset object.

Note about external buffers:
External buffers are not supported. When an sdTF buffer component contains a uri property to reference an external buffer data, the SDK throws an error when the user tries to access its data.

const sdk = await create()
const parser = await sdk.createParser()

const sdtfBuffer = new Uint8Array([ /* ... */ ])
const asset = parser.readFromBuffer(sdtfBuffer.buffer)

parser.readFromUrl:
The SDK fetches the sdTF file from the given URL and returns a new ISdtfReadableAsset object.

If possible, the SDK tries to acquire data via HTTP range requests to reduce network traffic. However, when the server does not support range requests, the full sdTF file is loaded.

Note about external buffers:
When an sdTF buffer component contains a uri property to reference an external buffer data, the SDK interprets the URI as a URL relative to the location of the parsed sdTF file, and tries to fetch the buffer when the user tries to access its data.

const sdk = await create()
const parser = await sdk.createParser()
const asset = await parser.readFromUrl("https://shapediver.com/sdtf")

parser.readFromFile:
The SDK reads the sdTF file at the specified path and returns a new ISdtfReadableAsset object.

Note about external buffers:
When an sdTF buffer component contains a uri property to reference an external buffer data, the SDK interprets the URI as a file path relative to the location of the parsed sdTF file, and tries to read the buffer file when the user tries to access its data.

Only available in Node.js!

const sdk = await create()
const parser = await sdk.createParser()
const asset = await parser.readFromFile("./test_data/sdTF_spec_example.sdtf")

Readable sdTF asset

The ISdtfParser returns a new ISdtfReadableAsset instance when a sdTF file is parsed, which holds the sdTF JSON content and provides functions to access the data. Like the sdTF JSON content, the ISdtfReadableAsset stores information in the form of one or more hierarchical tree structures. The following functions give access to the individual elements of the trees:

readableAsset.accessors:
Holds all accessor components of the sdTF file.
Accessors are used to link content data that is stored inside a buffer, and they can be referenced by data items and attributes.

readableAsset.attributes:
Holds all attributes components of the sdTF file.
Attributes are used to attach additional information to components, and they can be referenced by chunks, nodes and data items. They can either store content data directly in the JSON content of the sdTF, or they store the data in a binary buffer and reference it.

readableAsset.buffers:
Holds all buffer components of the sdTF file.
Buffers are used to story binary data, and are referenced by buffer views.

readableAsset.bufferViews:
Holds all buffer view components of the sdTF file.
Buffer views specify portions of a buffer, and are referenced by data items and attributes.

readableAsset.chunks:
Holds all chunk components of the sdTF file.
Chunks are fairly similar to nodes, but they provide the entry points to the hierarchical tree structure of sdTF.

readableAsset.items:
Holds all data item components of the sdTF file.
Data items represent leaf nodes in the hierarchical tree structure of sdTF. They can either store content data directly in the JSON content of the sdTF, or they store the data in a binary buffer and reference it.

readableAsset.nodes:
Holds all node components of the sdTF file.
Nodes represent nodes with children in the hierarchical tree structure of sdTF.

readableAsset.typeHints:
Holds all type hint components of the sdTF file.
Type hints are used to assign type information. When they are referenced by data items or attributes, they specify the type of their respective data content. However, when they are referenced by chunks or nodes, they indicate that all the data content hold by data items of these nodes and all their sub-nodes are of the referenced type.

readableAsset.fileInfo:
Holds the file info component of the sdTF file.
The file info contains general information about the sdTF asset.

Get data content

According to the sdTF v1 specification, data content is stored in data items and attributes, whereby their respective content type is specified via typehint components. This data content can either be stored directly in the JSON content of the sdTF file, or in the binary buffer. In both cases, the data content can be requested via the getContent() function in ISdtfReadableDataItem and ISdtfReadableAttribute. This function extracts the data value, and, when sdTF integrations have been registered in the SDK, processes it by applying integration-readers that support the respective type hint. The result is then returned to the callee.

const sdk = await create()
const parser = await sdk.createParser()

// Reads the example file given in the sdTF specification v1:
// https://github.com/shapediver/sdTF/tree/development/specification/1.0#a-complete-example
const asset = await parser.readFromFile("./test_data/sdTF_spec_example.sdtf")

// Item[1]:
// {
//     "accessor": 1,   // references the data in the binary buffer
//     "typeHint": 0    // references type "rhino.mesh"
// }
const value1 = await asset.items[1].getContent()  // `ISdtfBufferValue { id: 'e2bb8f80-5df3-41a4-b6ad-ce5e71f2bd06',  data: DataView }`

// Item[4]:
// {
//     "typeHint": 2,   // references type "double"
//     "value": 1       // stores data `1` directly in the JSON content
// }
const value4 = await asset.items[4].getContent()  // `1`

Note about data content stored in a buffer:
A single buffer view component might be used to store multiple objects. In this case, the accessor that references this buffer view also contains an id property. This ID can be used to reference individual objects inside the buffer view. Thus, when data content is loaded from a buffer, the result is wrapped in a ISdtfBufferValue object, that contains the id and the data content.

Note about external data content:
When data is stored in an external sdTF buffer component that contains a uri property, the SDK tries to load the external buffer before extracting the requested data content. This way, external sdTF buffers are loaded on-demand to improve the overall performance when reading a sdTF file.

Creating a new sdTF file

The ISdtfConstructor object exposes functionality to create a new sdTF file. A new sdTF structure can either be defined by creating individual sdTF components via a low-level factory and linking them manually, or by using a data-centric approach via a high-level writer interface.

constructor.getFactory:
Instantiates and returns a new ISdtfWriteableComponentFactory.
See creation via low-level factory for more information.

constructor.getWriter:
Instantiates and returns a new ISdtfWriter.
See creation via high-level writer for more information.

constructor.createBinarySdtf:
Both, the ISdtfWriteableComponentFactory and the ISdtfWriter, create a new ISdtfWriteableAsset object. This function creates a new sdTF file from the writeable-asset. During this step, the structure of the sdTF is optimized by complementing and merging duplicated type hints components, and merging buffer components. Additionally, when sdTF integrations have been registered in the SDK, the data content in all data items and attributes components is processed by integration-writers that support their respective type hint.

Creation via low-level factory

The ISdtfWriteableComponentFactory enables the callee to create new instances of writable sdTF components. These components can then be modified and linked as needed.

const sdk = await create()
const constructor = sdk.createConstructor()
const factory = constructor.getFactory()

// Create a data item with 2 attribute - both storing their content directly in the sdTF JSON content object
const dataItem1 = factory.createDataItem("foobar", "string")
dataItem1.attributes = factory.createAttributes({ "randomness1": [ Math.random(), "double" ] })
dataItem1.attributes = factory.createAttributes({ "randomness2": [ Math.random(), "double" ] })

// Creating a data item that stores the content in the sdTF binary buffer
const dataItem2 = factory.createDataItem({ data: new Uint8Array([ 115, 100, 116, 102 ]).buffer, contentType: "text" }, "data")

// Create a new chunk and add both data items
const chunk = factory.createChunk("root")
chunk.items.push(dataItem1, dataItem2)

// Create a new asset object (automatically adds default file info) and add the chunk
const asset = factory.createAsset()
asset.chunks.push(chunk)

// Creates a new sdTF file from the writeable-asset
const sdtf = constructor.createBinarySdtf(asset)

// sdTF - JSON content:
// {
//     "asset": {
//         "generator": "ShapeDiverSdtfWriter",
//         "version": "1.0"
//     },
//     "chunks": [
//         {
//             "items": [ 0, 1 ],
//             "name": "root"
//         }
//     ],
//     "nodes": [],
//     "items": [
//         {
//             "attributes": 0,
//             "typeHint": 0,
//             "value": "foobar"
//         },
//         {
//             "accessor": 0,
//             "typeHint": 1
//         }
//     ],
//     "attributes": [
//         {
//             "randomness1": {
//                 "typeHint": 2,
//                 "value": 0.8678168398187562
//             },
//             "randomness2": {
//                 "typeHint": 2,
//                 "value": 0.26541621248656133
//             }
//         }
//     ],
//     "typeHints": [
//         { "name": "string" },
//         { "name": "data" },
//         { "name": "double" }
//     ],
//     "accessors": [
//         { "bufferView": 0 }
//     ],
//     "bufferViews": [
//         {
//             "buffer": 0,
//             "byteLength": 4,
//             "byteOffset": 0,
//             "contentType": "text"
//         }
//     ],
//     "buffers": [ 
//         { "byteLength": 4 }
//    ]
// }

Creation via high-level writer

The ISdtfWriter enables more of a data-centric approach to create new sdTF files. Each available function creates a specific sdTF structure and fills it with the provided user data.

writer.createSimpleDataSdtf:
This function creates a simple, linear sdTF structure that consists of a single chunk with one or more data nodes. The given user data consists of data items and optional attributes.

const sdk = await create()
const constructor = sdk.createConstructor()

const asset = constructor.getWriter().createSimpleDataSdtf("tester", [
    // Creating two data items, both storing their content directly in the sdTF JSON content object
    { content: "foobar", typeHint: "string" },
    { content: Math.random(), typeHint: "double" },

    // Creating a data item that stores the content in the sdTF binary buffer
    { content: { data: new Uint8Array([ 115, 100, 116, 102 ]).buffer, contentType: "text" }, typeHint: "data" }
])

// Creates a new sdTF file from the writeable-asset
const sdtf = constructor.createBinarySdtf(asset)

// sdTF - JSON content:
// {
//     "asset": {
//       "generator": "ShapeDiverSdtfWriter",
//       "version": "1.0"
//     },
//     "chunks": [
//         {
//             "items": [ 0, 1, 2 ],
//             "name": "tester" 
//         }
//     ],
//     "nodes": [],
//     "items": [
//         {
//             "typeHint": 0,
//             "value": "foobar"
//         },
//         {
//             "typeHint": 1,
//             "value": 0.29872278297090205
//         },
//         {
//             "accessor": 0,
//             "typeHint": 2
//         }
//     ],
//     "attributes": [],
//     "typeHints": [
//         { "name": "string" },
//         { "name": "double" },
//         { "name": "data" }
//     ],
//     "accessors": [
//         { "bufferView": 0 }
//     ],
//     "bufferViews": [
//         {
//             "buffer": 0,
//             "byteLength": 4,
//             "byteOffset": 0,
//             "contentType": "text"
//         }
//     ],
//     "buffers": [
//         { "byteLength": 4 }
//     ]
// }

writer.createGrasshopperSdtfBuilder:
This function returns a builder instance to generate a structure that holds one or more Grasshopper® data trees.

A Data Tree in Grasshopper® is a hierarchical structure for storing data in nested lists. Every tree represents a single parameter and consists of branches and paths. While the branches store the actual data in sub-lists, every sub-list in paths corresponds to a branch name and is used to index it. Thus, the number of sub-lists in branches and paths must be equal.

Note about current restrictions:
Every tree can consist of only a single type of data (all type hints must be the same).

const sdk = await create()
const constructor = sdk.createConstructor()
const factory = constructor.getFactory()
const builder = constructor.getWriter().createGrasshopperSdtfBuilder()

// Create two branches that hold the data - all of the same type.
// It must consist of as many sub-lists as `paths.`
const branches = [
    [
        // Creating two data items, both storing their content directly in the sdTF JSON content object
        factory.createDataItem("foo", "string"),
        factory.createDataItem("bar", "string"),
    ],
    [
        // Creating a data item that stores the content directly in the sdTF JSON content object
        factory.createDataItem("baz", "string"),
    ],
]

// Create two paths, one for each branch.
// Note: "[ 0, 0 ]" is the name of the first branch, "[ 0, 1 ]" is the name of the seconds branch.
const paths = [
    [ 0, 0 ],
    [ 0, 1 ],
]

// Set the data of a single parameter.
// The parameter is represented by a new chunk that acts as an entry point for the data tree.
builder.addChunkForTreeData("ce0065af-8a90-4cd7-aa83-b8551fa7174a", { branches, paths })

// Create the asset
const asset = builder.build()

// Creates a new sdTF file from the writeable-asset
const sdtf = constructor.createBinarySdtf(asset)

// sdTF - JSON content:
// {
//     "asset": {
//         "generator": "ShapeDiverSdtfWriter",
//         "version": "1.0"
//     },
//     "chunks": [
//         {
//             "name": "ce0065af-8a90-4cd7-aa83-b8551fa7174a",
//             "nodes": [ 0, 1 ],
//             "typeHint": 0
//         }
//     ],
//     "nodes": [
//         {
//             "items": [ 0, 1 ],
//             "name": "[0,0]",
//             "typeHint": 0
//         },
//         {
//             "items": [ 2 ],
//             "name": "[0,1]",
//             "typeHint": 0
//         }
//     ],
//     "items": [
//         {
//             "typeHint": 0,
//             "value": "foo"
//         },
//         {
//             "typeHint": 0,
//             "value": "bar"
//         },
//         {
//             "typeHint": 0,
//             "value": "baz"
//         }
//     ],
//     "attributes": [],
//     "typeHints": [
//         { "name": "string" }
//     ],
//     "accessors": [],
//     "bufferViews": [],
//     "buffers": []
// }

Formatter

The ISdtfFormatter provides functionality to prettify the JSON content of an sdTF asset.

formatter.prettifyReadableAsset:
Generates the JSON content of the given readable-asset, prettifies it and returns the JSON as a string.

const sdk = await create()
const formatter = sdk.createFormatter()

const sdtf = await sdk.createParser().readFromFile("./test_data/sdTF_spec_example.sdtf")  // Creates a readable-asset
console.log(formatter.prettifyReadableAsset(sdtf))                                        // Logs the prettified JSON content to the console

formatter.prettifyWriteableAsset:
Generates the JSON content of the given writeable-asset. When sdTF integrations have been registered in the SDK, the data content in all data items and attributes components is processed by integration-writers that support their respective typehint. Afterwards, the structure of the generated JSON content is further optimized. The resulting JSON is then prettified and returned as a string.

const sdk = await create()
const writer = await sdk.createConstructor().getWriter()
const formatter = sdk.createFormatter()

const data: ISdtfWriterDataItem = { content: "foobar", typeHint: "string" }
const sdtf = writer.createSimpleDataSdtf("tester", [ data ])   // Creates a writeable-asset
console.log(formatter.prettifyWriteableAsset(sdtf))            // Logs the prettified JSON content to the console

CLI

This module comes with a simple, zero-overhead CLI tool to print out the JSON content of a specified sdTF file.

Examples:

# Fetch sdTF from URL
$ npx sdtf-v1 json-content -u "https://shapediver.com/sdtf"

# Read sdTF from file
$ npx sdtf-v1 json-content -f "./test_data/sdTF_spec_example.sdtf"

Support

If you have questions, please use the ShapeDiver Help Center.

You can find out more about ShapeDiver right here.

Licensing

This project is released under the MIT License.