tiny-json

This is a node module that takes an unique approach to packing JSON data. Perfect if you've got a lot of JSON objects with a similar structure!

It is said that 99.9% of human DNA is exactly the same in all people. The most efficient way of storing information about a million people must therefore be to store what they have in common (the 99.9%) in one place, and then all the million small things (0.1%) that makes us unique separate. You'll often find that JSON objects also have a lot in common, and this the idea behind tiny-json.

This is super useful if you want to save a lot a similar JSON data as strings in a key value database like Redis. You'll save a lot of space and have great performance improvments. Long strings makes Redis queries slow!

How it works

First define a pattern that all your JSON objects will match, the more specific you are the better compression you'll get. Anything that's valid JSON can be written as a pattern. Next use the pack and unpack methods together with your pattern to compress and uncompress your data. The result can then be converted into a string with JSON.stringify() and put in a key value database like Redis, while your pattern is stored in your application source.

1var tiny = require('tiny-json');
2
3var pattern = ['object', {
4  foo: ['array', ['number']],
5  bar: ['boolean']
6}];
7
8var firstData = {
9  foo: [1337, 42, 127],
10  bar: true
11};
12
13var secondData = {
14  foo: [17, 123],
15  bar: false
16};
17
18var firstPacked = tiny.pack(firstData, pattern);
19// [[1337,42,127],1]
20// 32 -> 17 chars: 47% compression!
21
22var secondPacked = tiny.pack(secondData, pattern);
23// [[17,123],0]
24// 28 -> 12 chars: 57% compression!
25
26var firstUnpacked = tiny.unpack(data, pattern);
27var secondUnpacked = tiny.unpack(data, pattern);

String compression is not fantastic but still pretty decent.

1var pattern = ['array',
2  ['object', {
3    id: ['number'],
4    description: ['string'],
5    status: ['enum', ['PENDING', 'CONFIRMED', 'REJECTED']]
6  }]
7];
8
9var data = [
10  { id: 17, description: 'this is pretty cool', status: 'REJECTED' },
11  { id: 12, description: 'one small step for man...', status: 'PENDING' },
12  { id: 15, description: 'okay lets get this party started', status: 'CONFIRMED' },
13];
14
15tiny.pack(data, pattern);
16// [[17,'஀낦Ӧ@阌��⮁豍勉Ǎꀐ턓ꀆ찀Ტ씤耀',2],
17//  [12,'㶃낦@츋悆ư琂㡀曀ᎅ㨒耀',0],
18//  [15,'㶆낆ৠЃ悦Ű㎔๨ꠠᘄ녀᳀ু梐墄저',1]]
19
20// 235 -> 81 chars: 66% compression!
21

Booleans and enums are pretty great though.

1var pattern = ['object', {
2  someBooleanValue: ['boolean'],
3  anotherBooleanValue: ['boolean'],
4  arrayOfBooleans: ['array', ['boolean']],
5  someEnum: ['enum', ['START', 'STOP', 'PAUSE']],
6  someOtherEnum: ['enum', ['COOL', 'AWESOME', 'WICKED']]
7}];
8
9var data = {
10  someBooleanValue: true,
11  anotherBooleanValue: false,
12  arrayOfBooleans: [true, false, false, true],
13  someEnum: 'PAUSE',
14  someOtherEnum: 'AWESOME'
15};
16
17tiny.pack(data, pattern);
18// [1,0,[1,0,0,1],2,1]
19
20// 140 -> 19 chars: 86% compression!

This is pretty pointless but it works.

1var pattern = ['number'];
2
3var data = 42;
4
5tiny.pack(data, pattern);
6// 42
7
8// 2 -> 2 chars: 0% compression! :P

You can store intermediate patterns so that you dont repeat yourself.

1var person = ['object', {
2  name: ['string'],
3  age: ['number'],
4  favoriteColor: ['enum', ['RED', 'GREEN', 'BLUE', 'PINK', 'YELLOW', 'ORANGE', 'PURPLE']]
5}];
6
7var pattern = {
8  mother: person,
9  father: person,
10  children: ['array', person]
11};
12
13var family = {
14  mother: { name: 'Jenny', age: 47, favoriteColor: 'PINK' },
15  father: { name: 'Bob', age: 50, favoriteColor: 'GREEN' },
16  children: [
17    { name: 'Kim', age: 19,  favoriteColor: 'BLUE' },
18    { name: 'Mikael', age: 15, favoriteColor: 'YELLOW' },
19    { name: 'Sara', age: 14, favoriteColor: 'PURPLE' }
20  ]
21};
22
23tiny.pack(family, pattern);
24// [['ᒅぶ悞䀀',47,3],
25//  ['ႇ끆䀀',50,1],
26//  [['㒄낶䀀',19,2],['Ⲅ냖ࡠꘃ搀',15,4],['㊄ぎ␀',14,6]]]
27
28// 278 -> 71 chars: 74% compression!
29

Full list of patterns

['object', { key: *pattern*, ... }] An object litteral with keys and values that are other patterns. Can be nested.

['array', *pattern*] An array with a pattern describing the elements in it. Can be nested, ex multi dimensional array of objects.

['boolean'] A boolean value, true or false.

['enum', *array of strings*] One of the strings provided in the array of strings, its convetion to use uppercase words.

['string'] A string, will be compressed with lz-string compression.

['number', *options*] A number, can take an optional object with the following keys: gt: number greater than, lh: number less than.

['any'] Any value, will not be compressed.

How to contribute

The source is well structured and easy to expand upon. If you come up with a way to compress a certain data type your contribution is welcome. Read this blog post written by David Hamp-Gonsalves for some great inspiration.

It's only a few simple steps.

1. Add a compress method:

1compress.something = function (value, pattern) {
2  // Compress the value in some smart way.
3  // If you pass options to your type
4  // those will be availible in the pattern.
5  return value;
6};

2. Add a uncompress method:

1uncompress.something = function (value, pattern) {
2  // Undo whatever you did to compress the value.
3  return value;
4};

3. Add a test for each of the methods:

1it('something', function () {
2  assert.strictEqual(tiny._compress('my value'), 'my value');
3});

4. Add the type to the "full list of patterns" in the README.

And thats it! Commit and push your changes and they will be merged asap.