JSON Type Decoders

Why

1. Decode plain JSON (of unknow type) into a typed data structure, or throw exception,
2. Concise & composable decoder definitions.
3. (Multiple) error messages showing where and why the parsing failed,
4. DRY: Auto-derives the resulting type from the decoder definition.
5. Returns a (deep) copy of the input.

Show me

1  import { decode, opt, string, boolean, def, number, alt, literalnull }  from "jsonTypeDecoder"
2
3  // Define a decoder for Foos by describing the 'shape' and how to decode the values:
4  const decodeFoo = decode({                // decode an object with...
5    foo: string,                            //  a string
6    somethingNested: {                      //  a nested object with
7      faz: opt(number),                     //    an optional number
8      fiz: [number],                        //    an array of numbers
9      foz: def(boolean, true),              //    a boolean, with a default value used if the field is missing.
10      fuz: alt(string, number, literalnull) //    either a string, number or a null. Tried in order.
11    },
12  })
13
14  // get the derived type of Foo (if needed).
15  type Foo = ReturnType<typeof decodeFoo>
16
17  //  type Foo = {
18  //    foo: string;
19  //    somethingNested: {
20  //        faz: number | undefined;
21  //        fiz: number[];
22  //        foz: boolean;
23  //        fuz: string | number | null;
24  //    };
25  //  }
26
27  // Use the decoder (with bad json)
28  const fooFail = decodeFoo(JSON.parse(
29    '{ "foo": true, "somethingNested": { "fiz" : [3,4,true,null], "foz": "true", "fuz": {} } }'
30  ))
31
32  // Exception Raised:
33  //   TypeError: Got a boolean (true) but was expecting a string at object.foo
34  //   TypeError: Got a boolean (true) but was expecting a number at object.somethingNested.fiz[2]
35  //   TypeError: Got a null but was expecting a number at object.somethingNested.fiz[3]
36  //   TypeError: Got a string ("true") but was expecting a boolean at object.somethingNested.foz
37  //   TypeError: Got an object but was expecting a string, a number or a null at object.somethingNested.fuz
38  //   in: {
39  //     "foo": true,
40  //     "somethingNested": {
41  //       "fiz": [
42  //         3,
43  //         4,
44  //         true,
45  //         null
46  //       ],
47  //       "foz": "true",
48  //       "fuz": {}
49  //     }
50  //   }

What else?

Sets, Maps, Dates, Tuples, Dictionary, numberString.

Transform plain JSON into richer TS data types.

1  const mammal = stringLiteral('cat', 'dog', 'cow') // decoders are functions that
2  type Mammal = ReturnType<typeof mammal>           //   are composable
3  // type Mammal = "cat" | "dog" | "cow"
4
5  const decodeBar = decode({                  // an object
6    bar: mammal,                              //   use an existing decoder
7    ber: literalValue(['one', 'two', 3]),     //   match one of the given values (or fail)
8    bir: set(mammal),                         //   converts JSON array into a JS Set<Mammal>
9    bor: map(number, tuple(string, date)),    //   date decodes epoch or full iso8601 string
10    bur: dict(isodate),                       //   decode JSON object of iso8601 strings...
11  }, { name: 'Foo' })                         // Name the decoder for error messages.
12
13  // Auto derived type of Bar
14  type Bar = ReturnType<typeof decodeBar>
15  //   type Bar = {
16  //     bar: "cat" | "dog" | "cow",
17  //     ber: string | number,
18  //     bir: Set<"cat" | "dog" | "cow">,
19  //     bor: Map<number, [string, Date]>,
20  //     bur: Dict<Date>,                     // ... into a Dict of JS Date objects
21  // }

The result of a decode can be anything: Date, Map, Set or a user defined type / class.

User Defined Functions

The decoded JSON can be transformed / validated / created with user functions

1
2  class Person { constructor(readonly name: string) { } }
3
4  const decodePap = decode({
5    pap: withDecoder([string], a => new Person(a.join(','))), // decode an array of strings, then transform into a Person
6    pep: decoder((u: unknown): string => {                    // wrap a user function into a combinator,
7      if (typeof (u) != 'boolean') { throw 'not a boolean' }  //   handling errors as needed.
8      return u ? 'success' : 'error'
9    }),
10    pip: validate(string, {                                   // use the decoder, then validate 
11      lengthGE3: s => s.length >= 3,                          //   against named validators.
12      lengthLE10: s => s.length <= 10,                        //   All validators have to be true.
13    }),
14  })
15
16  type Pap = ReturnType<typeof decodePap>
17  // type Pap = {
18  //   pap: Person;
19  //   pep: string;
20  //   pip: string;
21  // }
22
23  // Use the decoder (with bad json)
24  const papFail =  decodePap(JSON.parse(
25      '{"pap": ["one",2], "pep":"true","pip": "12345678901234" }'
26    ))
27
28  // Exception Raised:
29  // TypeError: Got a number (2) but was expecting a string at object.pap[1]
30  // DecodeError: UserDecoder threw: 'not a boolean' whilst decoding a string ("true") at object.pep
31  // DecodeError: validation failed (with: lengthLE10) whilst decoding a string ("12345678901234") at object.pip
32  // in: {
33  //   "pap": [
34  //     "one",
35  //     2
36  //   ],
37  //   "pep": "true",
38  //   "pip": "12345678901234"
39  // }

The numberString decoder converts a string to a number (including NaN, Infinity etc). Useful for decoding numbers from stringts (eg environment variables).

Dynamically choose Decoder to use.

The decoder can be selected at decode-time based on some aspect of the source JSON:

1  const decodeBSR = lookup('type', {                    // decode an object, get field named 'type' & lookup the decoder to use
2    body: {                                             // if the 'type' field === 'body' use the following decoder:
3      body: jsonValue,                                  //  deep copy of source JSON ensuring no non-Json constructs (eg Classes)
4      typeOfA: path('^.json.a', decoder(j => typeof j)) //  try a decoder at a different path in the source JSON.
5    },                                                  //      In this case adds a field to the output.
6    status: ternary(                                    // if the 'type' field === 'status'
7      { ver: 1 },                                       //  test that there is a 'ver' field with the value 1
8      { status: withDecoder(number, n => String(n)) },  //    'ver' === 1 : convert 'status' to a string.
9      { status: string },                               //    otherwise   : decode a string
10    ),
11    result: {                                           // if the 'type' field === 'result'
12      result: type({                                    //  decode the result field based on its type
13        number: n => n + 100,                           //    in all cases return a number
14        boolean: b => b ? 1 : 0,
15        string: s => Number(s),
16        array: a => a.length,
17        object: o => Object.keys(o).length,
18        null: constant(-1)                              //    ignore the provided value (null) and return -1
19      })
20    }
21  })
22
23  type BSR = ReturnType<typeof decodeBSR>
24
25  //  type ActualBSR = {
26  //      status: string;
27  //  } | {
28  //      body: JsonValue;
29  //      typeOfA: "string" | "number" | "bigint" | "boolean" | "symbol" | "undefined" | "object" | "function";
30  //  } | {
31  //      result: number;
32  //  }
33
34  console.log('res =', decodeBSR({ type: 'result', result:[200]}) );
35  // res = { result: 1 }

Check the derived Type against a 'known good' Type

Sometimes you may already have an existing type definition and need a decoder for it. Whilst you can't derive a decoder from a given type, you can check that the output of a decoder matches an existing type.

1
2  type ExpectedBSR = {                                  // Note that the 'type' is NOT in the derived type.
3    body: JsonValue;
4    typeOfA: "string" | "number" | "bigint" | "boolean" | "symbol" | "undefined" | "object" | "function"
5  } | {
6    status: string
7  } | {
8    result: number
9  }
10
11  // The line will fail to type check if the derived type of decodeBSR doenst match the provided type
12  // NOTE: The error messages can either be too vague or horrendous!
13  checkDecoder<ExpectedBSR, typeof decodeBSR>('')

literalValue vs. stringLiteral

1  const decoder1 = literalValue(['one','two',3])
2  // const decoder1: DecoderFn<string | number, any>
3
4  const decoder2 = stringLiteral('one','two','three')
5  // const decoder2: DecoderFn<"one" | "two" | "three", any>

A DecoderFn<OUT,IN> is an alias for (unknown: IN, ...args: any) => OUT - ie a function that returns a value of type OUT.

decoder1() returns type string | number if the source json is equal to any of the values in the argument to literalValue().

decoder2() returns the string literal type "one" | "two" | "three" if the source json is equal to any of the proveded arguments to stringLiteral().

array, object and decode

array(), as you'd expect, decodes an array of items that in turn have been decoded by the argument.

object() takes an object as argument where each property value is a decoder.

1const arrayOfStringsDecoder = array(string)
2// const arrayOfStringsDecoder: DecoderFn<string[], any>
3
4
5const arrayOfObjectsDecoder = object({ field1: string, field2: number })
6// const arrayOfObjectsDecoder: DecoderFn<{
7//   field1: string;
8//   field2: number;
9// }, any>

decode() transforms a structure ( of objects, arrays, & decoder functions ) into a decoder function. This is done by recursively descending into the structure replacing:

• objects -> with the object() decoder
• array -> with a combined array(alt()) decoder, NOT a tuple as you may be led to believe from the syntax ¹
• boolean / number / string / null -> literalValue() decoder
• decoder function -> decoder function (no change)

alt / every and family

The alt() family:

• alt(d1,d2,d3) : try the supplied decoders in turn until one succeeds.
• altT(tupleOfDecoders [, options]) : try the supplied decoders in turn until one succeeds.

The every() family:

• every(d1,d2,d3) : all the supplied decoders must succeed
• everyT(tupleOfDecoders [, options]) : all the supplied decoders must succeed.
• everyO(ObjectOfDecoders [, options]) : all the supplied decoders must succeed.

path

path(pathLocation,decoder) : move to another part of the source and try the decoder at that location. The pathLocation can either be string ( eg '^.field.names[index].etc', where ^ means traverse up the source), or an array of path components ( eg [UP, 'field', 'names', index, 'etc']). If the path cannot be followed, (eg field name into an array) then fail (unless the autoCreate option is set)

Options

There are a number of optoins that change the behaviour of some of the decoders, or the error messages that are generated on failure.

• name (string) : the name of the decoder
• ignoringErrors (boolean) : ignore decoding exceptions when decoding arrays, sets & maps
• noExtraProps (boolean) : check that an object doesn't contain any extra fields
• onlyOne (boolean) : strictly only One decoder should succeed in alt or altT
• keyPath (PathSpec) : the path of the key when using the map decoder (default: 'key')
• valuePath (PathSpec) : the path of the key when using the map decoder (default: 'value')
• autoCreate (boolean) : build deeply nested objects
• objectLike (boolean) : accept objects AND classes when decoding objects (eg process.env)

The map(keyDecoder,valueDecoder) decoder attempts to decode the following JSON stuctures into a Map<> type:

• object: the keys (strings) are passed into the keyDecoder and values passed into the valueDecoder,
• Array of [keyJSON,valueJSON] tuples.
• Array of objects, in which case use the provided paths to locate the key / values for each object in the Array.

Calling constructors or functions

Up till now, the decoders have been defined by describing the 'shape' of the source JSON, and the resulting type will be of the same shape (ish). Some exceptions:

• path() decoder : "hey, go look over there and bring back the result",
• withDecoder() : change a JSON value into something else,
• every*() : change a single JSON value into many things.

But sometimes you don't want the structure in the result, just the decoded value. Like when you want to use the value as a function / method / constructor argument.

1// Class definition
2class Zoo {
3
4  constructor(
5    private petType: string,
6    private petCount: number,
7    private extras: ('cat' | 'dog')[],
8  ) { }
9
10  public get salesPitch() {
11    return `${this.petCount} ${this.petType}s and [${this.extras.join(', ')}]`
12  }
13
14  // static method to decode the class
15  static decode = construct(                // does the 'new' stuff ...
16    Zoo,                                    // Class to construct
17    path('somewhere.deeply.nested', string),// 1st arg, a string at the given path
18    { petCount: number },                   // 2nd arg, a number from the location
19    { pets: array(stringLiteral('cat', 'dog')) }, // 3rd arg, an array of cats/dogs
20  )
21}

In the case of construct() and call(), the 'shape' of the arguments are used to describe where in the json a value is to be found, but it is not used in the result.

Finally

Parse, don’t validate : "... the difference between validation and parsing lies almost entirely in how information is preserved"

Footnotes

1. Typescript parses [1, 2, 'three'] with a type of (number|string)[], so the runtime behaviour is to model a decoder for that type. The way to coerce a tuple type in Typescript is [1, 2, 'three'] as const which is a) ugly b) implies immutability and c) I couldn't get it to work. If you need to decode a tuple, use tuple()! ↩