data-standard
v1.0.2
Published
The code in this repository is aimed at data pipeline engineers with the intent of sharing broadly if it is applicable to other projects. To that end no sensitive or proprietary information such as company names, trade secrets, or customer data will be co
Downloads
3
Readme
Data Standard
The standard.json
file contains a generalized JSON standard for describing the queryable data entities. It is backed by a JSON schema definition file schema.json
.
Structure of the Standard
The standard implements a Map like data structure. The data entities are the keys, and the values are objects with properties that can be used to retrieve the data entity. Every item must have two properties: sourceEndpoint
and queries
. Everything else is optional.
- sourceEndpoint is a string value of an http endpoint of the associated data entity.
- queries is an object map of various querystrings that can be concatenated with the sourceEndpoint to filter requests.
Make use of the reserved $comment
keyword to add comment notations to the standard wherever you see fit. Visit this Understanding JSON Schema page for more information on this reserved keyword.
Project Organization
All source files can be found in the src/ directory. Everything is implemented in TypeScript.
- Example implementations are located in the src/examples/ directory.
- Tests, implemented with Jest, are located in the src/tests/ directory.
- Utility files, such as getting a bearer token, can be found in the src/utils/ directory.
- Note: the src/utils/getBearerToken.ts file requires environment variables to be defined. Please see the Contributing section of this README on how to set those up.
Contributing
- Prerequisite: Node.js LTS version or later
- Install project dependencies:
npm i
- Create a
.env
file at the root of the project directory for environment variables
- Create five variables:
CLIENT_ID=""
CLIENT_SECRET=""
TENANT_ID=""
REDIRECT_URI=""
RESOURCE=""
- Fill in the empty strings with the respective variables from your Azure Active Directory application in Azure.
Examples
Examples are meant to demonstrate the standard in a more realistic manner. They can make use of the utility functions located in the utils/
directory and should be constrained to a single file.
Examples should be executed using ts-node
(included as a dependency of this project). Thus, the example file should be considered a script and not a module. The package.json contains a scripts
property where you can create an npm script for executing examples. Check out the example:adf
script as an example.
New scripts should follow the template:
"example:<Example Name>": "ts-node --project tsconfig.json src/examples/<Example Name>.ts"
Tests
Running tests is as easy as executing npm run test
. Tests are powered by Jest test runner.
When adding new tests, keep in mind that Jest utility types are included globally. This means you do not to explicitly import methods such as test
, describe
, and expect
.
The standard is validated by the test suite using the Ajv package (another JSON validator).
Coverage is enabled! Run npm run test -- --coverage
to see the output.