xschema

v1.0.0

Published

2 years ago

High performance and extensible data processing library, including schema builder, data validator, and sanitizer.

Downloads

0High
0Medium
0Low

kobalicek

data json metadata object sanitize sanity schema validation validator

xschema

The xschema library is a high performance data processing and validation library based on an extensible model/schema builder. It allows to build a schema that can be then used to process and validate any kind of JavaScript data (the root variable can be object, array, or any other primitive type). The library is designed for critical areas where the performance is important and even a minor overhead at validation side can cause service delays. The data validation and processing has been moved into extreme by using a JavaScript code-generator that generates the best possible data processing and validating functions for any user-defined schema. The most used JavaScript engines today have built-in JIT compiler so the code generated by xschema is then compiled by the VM into a machine code that will execute very fast and outperform all JavaScript data processing libraries that don't use such technique.

The performance is not the only aspect and feature offered by xschema. The library has been designed in a way that it should be very straightforward to define a schema, to reuse or inherit the existing one, and to create your own types that will extend the built-in functionality. There is no library that could satisfy all possible needs and use-cases so the possibility to extend the library is important. The library provides the most important types as full-featured built-ins.

The schema structure is always declarative and most of the schemas can be serialized back to JSON (xschema calls it a normalized JSON). The library also allows to associate a custom information called metadata with any field. Metadata is completely ignored by xschema library, but other tools can take advantage of it (for example you can associate a SQL table names with your entities and use them in your DB layer).

Additionally, xschema has several data processing options that help to deal with common problems like implementing data insertion, updating, deletion, and querying. Processing options can also be used to filter out objects' properties that are not defined (useful when extracting information from request's body or from more objects mixed together) and to accumulate all validation errors to have complete report of the validation.

Disclaimer

The xschema library has been designed to solve common, but also very specific problems. It's very fast and the support for metadata allows to simply extend it by new features. All built-in features are used in production and you will find many of them handy when implementing web services that do CRUD operations, because a single schema can be used to validate data that is inserted, updated, queried, or deleted. The library has been designed to be very fast, but is also very complete and configurable.

Introduction

the xschema library uses a declarative approach to build schemas, but it comes with its own syntax instead of relying on existing solutions like JSONSchema. The main reason for such move was to simplify the way schemas are defined by introducing shortcuts and directives that start with $ character. Shortcuts are used to simplify declaration of the most common concepts (for example an array of integers can be written as $type: "int[]") and directives are used to configure the type itself. Object's members are always defined without a $ prefix, but it's possible to define also members that start with $ by escaping it as \\$ (escaping and schema normalization is explained later).

var PersonSchema = xschema.schema({
  firstName  : { $type: "text", $maxLength: 64 },
  lastName   : { $type: "text", $maxLength: 64 },
  dateOfBirth: { $type: "date", $leapYear: false },

  active     : { $type: "bool" },
  score      : { $type: "int", $min: 0 },
  keywords   : { $type: "text[]" },
  bashrc     : { $type: "string", $maxLength: 4096 },

  address: {
    line1    : { $type: "text" },
    line2    : { $type: "text" },
    city     : { $type: "text" },
    zip      : { $type: "text" },
    country  : { $type: "text" }
  }
});

The example above defines a schema called PersonSchema, which is an object holding properties of various types specified by $type directive. Careful readers have noticed that the root object and nested address object have omitted the $type directive. It automatically uses object if no $type is provided, which allows to remove some verbosity in the schema declaration. Other directives like $min, $max, $leapYear, ..., are used to configure the type itself.

Confused by string vs text type? Well, string is any string in JavaScript in contrast to text, which is a string that doesn't contain \u0000-\u0008, \u000B-\u000C, and \u000E-\u001F characters. These characters have special meaning and in many cases their presence in your application's data is unwanted and may be dangerous.

Confused by [] suffix in keywords member? It's a xschema shortcut that defines an array, which can also be defined by using array type like this:

var KeywordsSchema = xschema.schema({
  $type: "array",
  $data: {
    $type: "text"
  }
});

var KeywordsSchema = xschema.schema({
  $type: "text[]"
});

Both schemas defined above are equivalent and internally normalized into the same structure.

Data Processing Concepts

The library comes with two base concepts that are used to work with data.

xschema.process(...) is a concept used to create a new data based on existing data. It's very useful in cases that more entities are mixed together in a single object and you need to separate/extract their content into independent objects. This happens for example in a request-body object. Data processing does not just validate the input data, but it can also sanity it before creating the output. If configured, you can trim/simplify input text, remove unknown properties, or insert fields having default values if they are not present.
xschema.test(...) is a concept used to test whether the given data conforms to the schema, but without using sanitizers.

Data Processing Options

Several data processing options exist that allow to use a single schema for multiple purposes. The default xschema.kNoOptions specifies no options and the schema is processed in a default way (i.e. strict mode).

The additional options are used to control:

Extraction Mode - Options xschema.kExtractTop, xschema.kExtractNested, and xschema.kExtractTop are used to control data extraction. Data extraction means extracting only specified properties from objects that can contain more properties that are not defined in the schema. It's useful when extracting parameters from a request body or to extract data that is known by schema, but without failing in cases that there is something that is not not known.
Delta Mode - Option xschema.kDeltaMode can be used to force nearly all properties to be optional. This is used in cases that you allow delta updates, but you still require some properties that specify DB keys to be present.
Error Accumulation - Option xschema.kAccumulateErrors is used in case that you want to get all errors that happened during data processing, but just the first one.

Data extraction options:

xschema.kExtractTop - Extract from top-level object only.
xschema.kExtractNested - Extract from nested object(s) only.
xschema.kExtractAll - Combination of xschema.kExtractTop and xschema.kExtractNested, which results in extraction from any "object".

TODO

Built-In Data Types

The following data types are built-in:

Any

Any $type is specified as any.

Any type directives:

Boolean

Boolean $type is specified as bool or boolean.

Boolean type directives:

Number and Integer

Number type $type is specified by the following type names and properties:

Type and Aliases | Minimum Value | Maximum Value | Description :----------------------- | :---------------- | :--------------- | :------------- double, number | None | None | Double precision floating point numeric | None | None | Numeric value (alias to double, but can be used to distinguish between double and numeric in case of describing DB schema) int8 | -128 | 127 | 8-bit signed integer uint8 | 0 | 255 | 8-bit unsigned integer int16, short | -32768 | 32767 | 16-bit signed integer uint16, ushort | 0 | 65535 | 16-bit unsigned integer int32 | -2147483648 | 2147483647 | 32-bit signed integer uint32 | 0 | 4294967295 | 32-bit unsigned integer int, integer | -9007199254740991 | 9007199254740991 | 53-bit signed integer, matches Number.isSafeInteger() behavior uint | 0 | 9007199254740991 | 53-bit unsigned integer, matches Number.isSafeInteger() behavior lat, latitude | -90 | 90 | Latitude (double precision) lon, longitude | -180 | 180 | Longitude (double precision)

Number type directives:

Character

Character $type is specified as char and it's a string that has length equal to one (that is, one character long string).

Character type directives:

String and Text

String $type is specified as string, text, or textline. If type string is specified any JavaScript string passes, however, if text type is specified the validator only passes if the string doesn't contain \u0000-\u0008, \u000B-\u000C, and \u000E-\u001F characters. Use text to disallow these characters that have special meaning and are in most cases unwanted (especially the \u0000 character). The textline type restricts text from using line and paragraph delimiter characters.

String/Text type directives:

BigInt

BigInt $type is specified as bigint. BigInt is a string that contains only ASCII digits (characters from 0 to 9) and an optional minus sign at the beginning. It allows to validate whether the number represented as a string doesn't overflow 64 bits and also allows to set a possible minimum and maximum value (also as string). BigInt can also be configured to allow more than 64-bits by using $min and $max directives, described below.

BigInt type directives:

Optionally, you can use xschema.misc.isBigInt(s, min, max) function to check whether a string value matches BigInt with optional min and max constraints. This function doesn't require a schema instance.

Color

Color $type is specified as color. Color is a string that matches #RGB, #RRGGBB or color-name format. It supports all color names that are defined by CSS specification and allows to include a dictionary having extra color names that you need to allow. Color names are case-insensitive by default.

Color type directives:

Credit Card

Credit card $type is specified as creditcard. It checks whether the string is a valid credit card number by using a LUHN algorithm. The validator doesn't accept dashes or any other characters used as separators.

Credit card type directives:

ISBN

ISBN $type is specified as isbn. It checks whether the string is a valid ISBN number.

ISBN type directives:

MAC Address

MAC address $type is specified as mac. MAC address is a string in form XX:XX:XX:XX:XX:XX that specifies a network MAC address.

MAC address type directives:

IP Address

IP address $type is specified as ip, ipv4, or ipv6. IP address is a string specifying a network IP address. The ip type matches both IPV4 and IPV6 addresses, in contrast to ipv4 and ipv6 types, that match IPV4 and IPV6 addresses, respectively.

IP address type directives:

UUID

UUID $type is specified as uuid. UUID validator is used to check whether the string contains a valid UUID number, that can be optionally surrounded by curly braces.

UUID type directives:

DateTime

DateTime $type is specified as date, datetime, datetime-ms, and datetime-us. It's a formatted string that contains date, time, or date+time components. The validator extracts these components and validates whether they are correct. Leap years and leap seconds support is built-in and can be configured through directives.

DateTime defaults:

DateTime format options:

Format Option |Fixed Length| Range | Description :----------------------- | :----------| :-------------- | :--------------------- Y | false | 1-9999 | Year (1-4 digits) YY | true | 00-99 | Year (2 digits) YYYY | true | 0001-9999 | Year (4 digits) M | false | 1-12 | Month (1-2 digits) MM | true | 01-12 | Month (2 digits) D | false | 1-31 | Day (1-2 digits) DD | true | 01-31 | Day (2 digits) H | false | 0-23 | Hour (1-2 digits) HH | true | 00-23 | Hour (2 digits) m | false | 0-59 | Minute (1-2 digits) mm | true | 00-59 | Minute (2 digits) s | false | 0-60 | Second (1-2 digits) ss | true | 00-60 | Second (2 digits) SSS | true | 000-999 | Millisecond (3 digits) SSSSSS | true | 000000-999999 | Microsecond (6 digits) ? | true | | Any other character requires exact match of that character, for example -, /, ., ,, etc...

DateTime type directives:

Map

Map $type is specified as map. It's an object where all keys are strings and all values have the same type specified by $data directive.

Map type directives:

Object

Object $type is specified as object or can be omitted completely. Object is a special type that allows to specify its members by using unprefixed keys (keys that don't start with "$"). For example the following specifies an object that has mandatory members (keys) "a" and "b":

var Schema = xschema({
  a: { $type: "int" },
  b: { $type: "int", $optional: true }
});

// These will pass.
xschema.test({ a: 1       }, Schema);
xschema.test({ a: 1, b: 2 }, Schema);

Object's directives and members can be mixed in the same definition, for example the following schema defines an object that has a nested object, which is optional:

var Schema = xschema({
  nested: {
    $optional: true,

    a: { $type: "int" },
    b: { $type: "int", $optional: true }
  }
});

// These will pass.
xschema.test({                        }, Schema);
xschema.test({ nested: { a: 1       } }, Schema);
xschema.test({ nested: { a: 1, b: 2 } }, Schema);

TODO

Object type directives:

Array

Array $type is specified as array or by [] suffix (shortcut).

Array type directives:

Custom Types

TODO

Global Directives

Global directives can be applied to any $type:

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

xschema

Disclaimer

Introduction

Data Processing Concepts

Data Processing Options

Built-In Data Types

Any

Boolean

Number and Integer

Character

String and Text

BigInt

Color

Credit Card

ISBN

MAC Address

IP Address

UUID

DateTime

Map

Object

Array

Custom Types

Global Directives