Data Modeling Support

Datastore abstraction for core operations.

Install: @travetto/model

npm install @travetto/model

# or

yarn add @travetto/model

This module provides a set of contracts/interfaces to data model persistence, modification and retrieval. This module builds heavily upon the Schema, which is used for data model validation.

Contracts

The module is mainly composed of contracts. The contracts define the expected interface for various model patterns. The primary contracts are Basic, CRUD, Indexed, Expiry, Blob and Bulk.

Basic

All Data Modeling Support implementations, must honor the Basic contract to be able to participate in the model ecosystem. This contract represents the bare minimum for a model service.

Code: Basic Contract

export interface ModelBasicSupport<C = unknown> {
  /**
   * Get underlying client
   */
  get client(): C;

  /**
   * Get by Id
   * @param id The identifier of the document to retrieve
   * @throws {NotFoundError} When an item is not found
   */
  get<T extends ModelType>(cls: Class<T>, id: string): Promise<T>;

  /**
   * Create new item
   * @param item The document to create
   * @throws {ExistsError} When an item with the provided id already exists
   */
  create<T extends ModelType>(cls: Class<T>, item: OptionalId<T>): Promise<T>;

  /**
   * Delete an item
   * @param id The id of the document to delete
   * @throws {NotFoundError} When an item is not found
   */
  delete<T extends ModelType>(cls: Class<T>, id: string): Promise<void>;
}

CRUD

The CRUD contract, builds upon the basic contract, and is built around the idea of simple data retrieval and storage, to create a foundation for other services that need only basic support. The model extension in Authentication, is an example of a module that only needs create, read and delete, and so any implementation of Data Modeling Support that honors this contract, can be used with the Authentication model extension.

Code: Crud Contract

export interface ModelCrudSupport extends ModelBasicSupport {

  /**
   * Id Source
   */
  idSource: ModelIdSource;

  /**
   * Update an item
   * @param item The document to update.
   * @throws {NotFoundError} When an item is not found
   */
  update<T extends ModelType>(cls: Class<T>, item: T): Promise<T>;

  /**
   * Create or update an item
   * @param item The document to upsert
   * @param view The schema view to validate against
   */
  upsert<T extends ModelType>(cls: Class<T>, item: OptionalId<T>): Promise<T>;

  /**
   * Update partial, respecting only top level keys.
   *
   * When invoking this method, any top level keys that are null/undefined are treated as removals/deletes.  Any properties
   * that point to sub objects/arrays are treated as wholesale replacements.
   *
   * @param id The document identifier to update
   * @param item The document to partially update.
   * @param view The schema view to validate against
   * @throws {NotFoundError} When an item is not found
   */
  updatePartial<T extends ModelType>(cls: Class<T>, item: Partial<T> & { id: string }, view?: string): Promise<T>;

  /**
   * List all items
   */
  list<T extends ModelType>(cls: Class<T>): AsyncIterable<T>;
}

Indexed

Additionally, an implementation may support the ability for basic Indexed queries. This is not the full featured query support of Data Model Querying, but allowing for indexed lookups. This does not support listing by index, but may be added at a later date.

Code: Indexed Contract

export interface ModelIndexedSupport extends ModelBasicSupport {
  /**
   * Get entity by index as defined by fields of idx and the body fields
   * @param cls The type to search by
   * @param idx The index name to search against
   * @param body The payload of fields needed to search
   */
  getByIndex<T extends ModelType>(cls: Class<T>, idx: string, body: DeepPartial<T>): Promise<T>;

  /**
   * Delete entity by index as defined by fields of idx and the body fields
   * @param cls The type to search by
   * @param idx The index name to search against
   * @param body The payload of fields needed to search
   */
  deleteByIndex<T extends ModelType>(cls: Class<T>, idx: string, body: DeepPartial<T>): Promise<void>;

  /**
   * List entity by ranged index as defined by fields of idx and the body fields
   * @param cls The type to search by
   * @param idx The index name to search against
   * @param body The payload of fields needed to search
   */
  listByIndex<T extends ModelType>(cls: Class<T>, idx: string, body?: DeepPartial<T>): AsyncIterable<T>;

  /**
   * Upsert by index, allowing the index to act as a primary key
   * @param cls The type to create for
   * @param idx The index name to use
   * @param body The document to potentially store
   */
  upsertByIndex<T extends ModelType>(cls: Class<T>, idx: string, body: OptionalId<T>): Promise<T>;
}

Expiry

Certain implementations will also provide support for automatic Expiry of data at runtime. This is extremely useful for temporary data as, and is used in the Caching module for expiring data accordingly.

Code: Expiry Contract

export interface ModelExpirySupport extends ModelCrudSupport {
  /**
   * Delete all expired by class
   *
   * @returns Returns the number of documents expired
   */
  deleteExpired<T extends ModelType>(cls: Class<T>): Promise<number>;
}

Blob

Some implementations also allow for the ability to read/write binary data as Blob. Given that all implementations can store Base64 encoded data, the key differentiator here, is native support for streaming data, as well as being able to store binary data of significant sizes.

Code: Blob Contract

export interface ModelBlobSupport {

  /**
   * Upsert blob to storage
   * @param location The location of the blob
   * @param input The actual blob to write
   * @param meta Additional metadata to store with the blob
   * @param overwrite Should we replace content if already found, defaults to true
   */
  upsertBlob(location: string, input: BinaryInput, meta?: BlobMeta, overwrite?: boolean): Promise<void>;

  /**
   * Get blob from storage
   * @param location The location of the blob
   */
  getBlob(location: string, range?: ByteRange): Promise<Blob>;

  /**
   * Get metadata for blob
   * @param location The location of the blob
   */
  getBlobMeta(location: string): Promise<BlobMeta>;

  /**
   * Delete blob by location
   * @param location The location of the blob
   */
  deleteBlob(location: string): Promise<void>;

  /**
   * Update blob metadata
   * @param location The location of the blob
   */
  updateBlobMeta(location: string, meta: BlobMeta): Promise<void>;

  /**
   * Produces an externally usable URL for sharing limited read access to a specific resource
   *
   * @param location The asset location to read from
   * @param exp Expiry
   */
  getBlobReadUrl?(location: string, exp?: TimeSpan): Promise<string>;

  /**
   * Produces an externally usable URL for sharing allowing direct write access
   *
   * @param location The asset location to write to
   * @param meta The metadata to associate with the final asset
   * @param exp Expiry
   */
  getBlobWriteUrl?(location: string, meta: BlobMeta, exp?: TimeSpan): Promise<string>;
}

Bulk

Finally, there is support for Bulk operations. This is not to simply imply issuing many commands at in parallel, but implementation support for an atomic/bulk operation. This should allow for higher throughput on data ingest, and potentially for atomic support on transactions.

Code: Bulk Contract

export interface ModelBulkSupport extends ModelCrudSupport {
  processBulk<T extends ModelType>(cls: Class<T>, operations: BulkOp<T>[]): Promise<BulkResponse>;
}

Declaration

Models are declared via the @Model decorator, which allows the system to know that this is a class that is compatible with the module. The only requirement for a model is the ModelType

Code: ModelType

export interface ModelType {
  /**
   * Unique identifier.
   *
   * If not provided, will be computed on create
   */
  id: string;
}

The id is the only required field for a model, as this is a hard requirement on naming and type. This may make using existing data models impossible if types other than strings are required. Additionally, the type field, is intended to record the base model type, but can be remapped. This is important to support polymorphism, not only in Data Modeling Support, but also in Schema.

Implementations

|Service|Basic|CRUD|Indexed|Expiry|Blob|Bulk| |-------|-----|----|-------|------|----|----| |DynamoDB Model Support|X|X|X|X| | | |Elasticsearch Model Source|X|X|X|X| |X| |Firestore Model Support|X|X|X| | | | |MongoDB Model Support|X|X|X|X|X|X| |Redis Model Support|X|X|X|X| || |S3 Model Support|X|X| |X|X| | |SQL Model Service|X|X|X|X| |X| |Memory Model Support|X|X|X|X|X|X| |File Model Support|X|X| |X|X|X|

Custom Model Service

In addition to the provided contracts, the module also provides common utilities and shared test suites. The common utilities are useful for repetitive functionality, that is unable to be shared due to not relying upon inheritance (this was an intentional design decision). This allows for all the Data Modeling Support implementations to completely own the functionality and also to be able to provide additional/unique functionality that goes beyond the interface. Memory Model Support serves as a great example of what a full featured implementation can look like.

To enforce that these contracts are honored, the module provides shared test suites to allow for custom implementations to ensure they are adhering to the contract's expected behavior.

Code: Memory Service Test Configuration

import { DependencyRegistry } from '@travetto/di';
import { AppError, castTo, Class, classConstruct } from '@travetto/runtime';

import { isBulkSupported, isCrudSupported } from '../../src/internal/service/common';
import { ModelType } from '../../src/types/model';
import { ModelSuite } from './suite';

type ServiceClass = { serviceClass: { new(): unknown } };

@ModelSuite()
export abstract class BaseModelSuite<T> {

  static ifNot(pred: (svc: unknown) => boolean): (x: unknown) => Promise<boolean> {
    return async (x: unknown) => !pred(classConstruct(castTo<ServiceClass>(x).serviceClass));
  }

  serviceClass: Class<T>;
  configClass: Class;

  async getSize<U extends ModelType>(cls: Class<U>): Promise<number> {
    const svc = (await this.service);
    if (isCrudSupported(svc)) {
      let i = 0;
      for await (const __el of svc.list(cls)) {
        i += 1;
      }
      return i;
    } else {
      throw new AppError(`Size is not supported for this service: ${this.serviceClass.name}`);
    }
  }

  async saveAll<M extends ModelType>(cls: Class<M>, items: M[]): Promise<number> {
    const svc = await this.service;
    if (isBulkSupported(svc)) {
      const res = await svc.processBulk(cls, items.map(x => ({ insert: x })));
      return res.counts.insert;
    } else if (isCrudSupported(svc)) {
      const out: Promise<M>[] = [];
      for (const el of items) {
        out.push(svc.create(cls, el));
      }
      await Promise.all(out);
      return out.length;
    } else {
      throw new Error('Service does not support crud operations');
    }
  }

  get service(): Promise<T> {
    return DependencyRegistry.getInstance(this.serviceClass);
  }

  async toArray<U>(src: AsyncIterable<U> | AsyncGenerator<U>): Promise<U[]> {
    const out: U[] = [];
    for await (const el of src) {
      out.push(el);
    }
    return out;
  }
}

CLI - model:export

The module provides the ability to generate an export of the model structure from all the various @Models within the application. This is useful for being able to generate the appropriate files to manually create the data schemas in production.

Terminal: Running model export

$ trv model:export --help

Usage: model:export [options] <provider:string> <models...:string>

Options:
  -e, --env <string>     Application environment
  -m, --module <module>  Module to run for
  -h, --help             display help for command

Providers
--------------------
  * SQL

Models
--------------------
  * samplemodel

CLI - model:install

The module provides the ability to install all the various @Models within the application given the current configuration being targeted. This is useful for being able to prepare the datastore manually.

Terminal: Running model install

$ trv model:install --help

Usage: model:install [options] <provider:string> <models...:string>

Options:
  -e, --env <string>     Application environment
  -m, --module <module>  Module to run for
  -h, --help             display help for command

Providers
--------------------
  * SQL

Models
--------------------
  * samplemodel