JSON Type Decoders

Why

1. Decode plain JSON (of unknow type) into a typed data structure, or throw exception,
2. Concise & composable decoder definitions.
3. (Multiple) error messages showing where and why the parsing failed,
4. DRY: Auto-derives the resulting type from the decoder definition.
5. Returns a (deep) copy of the input.

Show me

  import { decode, opt, string, boolean, def, number, alt, literalnull }  from "jsonTypeDecoder"

  // Define a decoder for Foos by describing the 'shape' and how to decode the values:
  const decodeFoo = decode({                // decode an object with...
    foo: string,                            //  a string
    somethingNested: {                      //  a nested object with
      faz: opt(number),                     //    an optional number
      fiz: [number],                        //    an array of numbers
      foz: def(boolean, true),              //    a boolean, with a default value used if the field is missing.
      fuz: alt(string, number, literalnull) //    either a string, number or a null. Tried in order.
    },
  })

  // get the derived type of Foo (if needed).
  type Foo = ReturnType<typeof decodeFoo>

  //  type Foo = {
  //    foo: string;
  //    somethingNested: {
  //        faz: number | undefined;
  //        fiz: number[];
  //        foz: boolean;
  //        fuz: string | number | null;
  //    };
  //  }

  // Use the decoder (with bad json)
  const fooFail = decodeFoo(JSON.parse(
    '{ "foo": true, "somethingNested": { "fiz" : [3,4,true,null], "foz": "true", "fuz": {} } }'
  ))

  // Exception Raised:
  //   TypeError: Got a boolean (true) but was expecting a string at object.foo
  //   TypeError: Got a boolean (true) but was expecting a number at object.somethingNested.fiz[2]
  //   TypeError: Got a null but was expecting a number at object.somethingNested.fiz[3]
  //   TypeError: Got a string ("true") but was expecting a boolean at object.somethingNested.foz
  //   TypeError: Got an object but was expecting a string, a number or a null at object.somethingNested.fuz
  //   in: {
  //     "foo": true,
  //     "somethingNested": {
  //       "fiz": [
  //         3,
  //         4,
  //         true,
  //         null
  //       ],
  //       "foz": "true",
  //       "fuz": {}
  //     }
  //   }

What else?

Sets, Maps, Dates, Tuples, Dictionary, numberString.

Transform plain JSON into richer TS data types.

  const mammal = stringLiteral('cat', 'dog', 'cow') // decoders are functions that
  type Mammal = ReturnType<typeof mammal>           //   are composable
  // type Mammal = "cat" | "dog" | "cow"

  const decodeBar = decode({                  // an object
    bar: mammal,                              //   use an existing decoder
    ber: literalValue(['one', 'two', 3]),     //   match one of the given values (or fail)
    bir: set(mammal),                         //   converts JSON array into a JS Set<Mammal>
    bor: map(number, tuple(string, date)),    //   date decodes epoch or full iso8601 string
    bur: dict(isodate),                       //   decode JSON object of iso8601 strings...
  }, { name: 'Foo' })                         // Name the decoder for error messages.

  // Auto derived type of Bar
  type Bar = ReturnType<typeof decodeBar>
  //   type Bar = {
  //     bar: "cat" | "dog" | "cow",
  //     ber: string | number,
  //     bir: Set<"cat" | "dog" | "cow">,
  //     bor: Map<number, [string, Date]>,
  //     bur: Dict<Date>,                     // ... into a Dict of JS Date objects
  // }

The result of a decode can be anything: Date, Map, Set or a user defined type / class.

User Defined Functions

The decoded JSON can be transformed / validated / created with user functions

  class Person { constructor(readonly name: string) { } }

  const decodePap = decode({
    pap: withDecoder([string], a => new Person(a.join(','))), // decode an array of strings, then transform into a Person
    pep: decoder((u: unknown): string => {                    // wrap a user function into a combinator,
      if (typeof (u) != 'boolean') { throw 'not a boolean' }  //   handling errors as needed.
      return u ? 'success' : 'error'
    }),
    pip: validate(string, {                                   // use the decoder, then validate 
      lengthGE3: s => s.length >= 3,                          //   against named validators.
      lengthLE10: s => s.length <= 10,                        //   All validators have to be true.
    }),
  })

  type Pap = ReturnType<typeof decodePap>
  // type Pap = {
  //   pap: Person;
  //   pep: string;
  //   pip: string;
  // }

  // Use the decoder (with bad json)
  const papFail =  decodePap(JSON.parse(
      '{"pap": ["one",2], "pep":"true","pip": "12345678901234" }'
    ))

  // Exception Raised:
  // TypeError: Got a number (2) but was expecting a string at object.pap[1]
  // DecodeError: UserDecoder threw: 'not a boolean' whilst decoding a string ("true") at object.pep
  // DecodeError: validation failed (with: lengthLE10) whilst decoding a string ("12345678901234") at object.pip
  // in: {
  //   "pap": [
  //     "one",
  //     2
  //   ],
  //   "pep": "true",
  //   "pip": "12345678901234"
  // }

The numberString decoder converts a string to a number (including NaN, Infinity etc). Useful for decoding numbers from stringts (eg environment variables).

Dynamically choose Decoder to use.

The decoder can be selected at decode-time based on some aspect of the source JSON:

  const decodeBSR = lookup('type', {                    // decode an object, get field named 'type' & lookup the decoder to use
    body: {                                             // if the 'type' field === 'body' use the following decoder:
      body: jsonValue,                                  //  deep copy of source JSON ensuring no non-Json constructs (eg Classes)
      typeOfA: path('^.json.a', decoder(j => typeof j)) //  try a decoder at a different path in the source JSON.
    },                                                  //      In this case adds a field to the output.
    status: ternary(                                    // if the 'type' field === 'status'
      { ver: 1 },                                       //  test that there is a 'ver' field with the value 1
      { status: withDecoder(number, n => String(n)) },  //    'ver' === 1 : convert 'status' to a string.
      { status: string },                               //    otherwise   : decode a string
    ),
    result: {                                           // if the 'type' field === 'result'
      result: type({                                    //  decode the result field based on its type
        number: n => n + 100,                           //    in all cases return a number
        boolean: b => b ? 1 : 0,
        string: s => Number(s),
        array: a => a.length,
        object: o => Object.keys(o).length,
        null: constant(-1)                              //    ignore the provided value (null) and return -1
      })
    }
  })

  type BSR = ReturnType<typeof decodeBSR>

  //  type ActualBSR = {
  //      status: string;
  //  } | {
  //      body: JsonValue;
  //      typeOfA: "string" | "number" | "bigint" | "boolean" | "symbol" | "undefined" | "object" | "function";
  //  } | {
  //      result: number;
  //  }

  console.log('res =', decodeBSR({ type: 'result', result:[200]}) );
  // res = { result: 1 }

Check the derived Type against a 'known good' Type

Sometimes you may already have an existing type definition and need a decoder for it. Whilst you can't derive a decoder from a given type, you can check that the output of a decoder matches an existing type.

  type ExpectedBSR = {                                  // Note that the 'type' is NOT in the derived type.
    body: JsonValue;
    typeOfA: "string" | "number" | "bigint" | "boolean" | "symbol" | "undefined" | "object" | "function"
  } | {
    status: string
  } | {
    result: number
  }

  // The line will fail to type check if the derived type of decodeBSR doenst match the provided type
  // NOTE: The error messages can either be too vague or horrendous!
  checkDecoder<ExpectedBSR, typeof decodeBSR>('')

literalValue vs. stringLiteral

  const decoder1 = literalValue(['one','two',3])
  // const decoder1: DecoderFn<string | number, any>

  const decoder2 = stringLiteral('one','two','three')
  // const decoder2: DecoderFn<"one" | "two" | "three", any>

A DecoderFn<OUT,IN> is an alias for (unknown: IN, ...args: any) => OUT - ie a function that returns a value of type OUT.

decoder1() returns type string | number if the source json is equal to any of the values in the argument to literalValue().

decoder2() returns the string literal type "one" | "two" | "three" if the source json is equal to any of the proveded arguments to stringLiteral().

array, object and decode

array(), as you'd expect, decodes an array of items that in turn have been decoded by the argument.

object() takes an object as argument where each property value is a decoder.

const arrayOfStringsDecoder = array(string)
// const arrayOfStringsDecoder: DecoderFn<string[], any>


const arrayOfObjectsDecoder = object({ field1: string, field2: number })
// const arrayOfObjectsDecoder: DecoderFn<{
//   field1: string;
//   field2: number;
// }, any>

decode() transforms a structure ( of objects, arrays, & decoder functions ) into a decoder function. This is done by recursively descending into the structure replacing:

• objects -> with the object() decoder
• array -> with a combined array(alt()) decoder, NOT a tuple as you may be led to believe from the syntax [^1]
• boolean / number / string / null -> literalValue() decoder
• decoder function -> decoder function (no change)

[^1]: Typescript parses [1, 2, 'three'] with a type of (number|string)[], so the runtime behaviour is to model a decoder for that type. The way to coerce a tuple type in Typescript is [1, 2, 'three'] as const which is a) ugly b) implies immutability and c) I couldn't get it to work. If you need to decode a tuple, use tuple()!

alt / every and family

The alt() family:

• alt(d1,d2,d3) : try the supplied decoders in turn until one succeeds.
• altT(tupleOfDecoders [, options]) : try the supplied decoders in turn until one succeeds.

The every() family:

• every(d1,d2,d3) : all the supplied decoders must succeed
• everyT(tupleOfDecoders [, options]) : all the supplied decoders must succeed.
• everyO(ObjectOfDecoders [, options]) : all the supplied decoders must succeed.

path

path(pathLocation,decoder) : move to another part of the source and try the decoder at that location. The pathLocation can either be string ( eg '^.field.names[index].etc', where ^ means traverse up the source), or an array of path components ( eg [UP, 'field', 'names', index, 'etc']). If the path cannot be followed, (eg field name into an array) then fail (unless the autoCreate option is set)

Options

There are a number of optoins that change the behaviour of some of the decoders, or the error messages that are generated on failure.

• name (string) : the name of the decoder
• ignoringErrors (boolean) : ignore decoding exceptions when decoding arrays, sets & maps
• noExtraProps (boolean) : check that an object doesn't contain any extra fields
• onlyOne (boolean) : strictly only One decoder should succeed in alt or altT
• keyPath (PathSpec) : the path of the key when using the map decoder (default: 'key')
• valuePath (PathSpec) : the path of the key when using the map decoder (default: 'value')
• autoCreate (boolean) : build deeply nested objects
• objectLike (boolean) : accept objects AND classes when decoding objects (eg process.env)

The map(keyDecoder,valueDecoder) decoder attempts to decode the following JSON stuctures into a Map<> type:

• object: the keys (strings) are passed into the keyDecoder and values passed into the valueDecoder,
• Array of [keyJSON,valueJSON] tuples.
• Array of objects, in which case use the provided paths to locate the key / values for each object in the Array.

Calling constructors or functions

Up till now, the decoders have been defined by describing the 'shape' of the source JSON, and the resulting type will be of the same shape (ish). Some exceptions:

• path() decoder : "hey, go look over there and bring back the result",
• withDecoder() : change a JSON value into something else,
• every*() : change a single JSON value into many things.

But sometimes you don't want the structure in the result, just the decoded value. Like when you want to use the value as a function / method / constructor argument.

// Class definition
class Zoo {

  constructor(
    private petType: string,
    private petCount: number,
    private extras: ('cat' | 'dog')[],
  ) { }

  public get salesPitch() {
    return `${this.petCount} ${this.petType}s and [${this.extras.join(', ')}]`
  }

  // static method to decode the class
  static decode = construct(                // does the 'new' stuff ...
    Zoo,                                    // Class to construct
    path('somewhere.deeply.nested', string),// 1st arg, a string at the given path
    { petCount: number },                   // 2nd arg, a number from the location
    { pets: array(stringLiteral('cat', 'dog')) }, // 3rd arg, an array of cats/dogs
  )
}

In the case of construct() and call(), the 'shape' of the arguments are used to describe where in the json a value is to be found, but it is not used in the result.

Finally

Parse, don’t validate : "... the difference between validation and parsing lies almost entirely in how information is preserved"