Superbuffer

A simple way to serialize JavaScript objects into ArrayBuffers for high compression on the wire.

Introduction • Usage • Data Types • Caveats

---

Introduction

Superbuffer makes it easy to serialize and deserialize data from ArrayBuffers matching the interfaces you define. Its Schema-Model API was inspired by typed-array-buffer-schema.

Superbuffer was designed to balance the readability and flexibility JSON while maximizing compression of your data on the wire. It is especially useful in situations where a high volume and frequency of network traffic between clients is necessary (such as multiplayer networked games).

Installation

NPM

npm i superbuffer

Yarn

yarn add superbuffer

Usage API

Superbuffer allows you to define the shape of the data you want to serialize:

1import {Schema, Model, uint32, uint64, int16, string} from 'superbuffer';
2
3const commentSchema = new Schema({
4  message: string,
5  timestamp: uint64,
6  votes: int16,
7});
8
9const postSchema = new Schema({
10  id: uint32,
11  title: string,
12  tags: [string],
13  author: {
14    id: uint32,
15    name: string,
16  },
17  comments: [commentSchema],
18});
19
20const post = new Model(postSchema);

If you use Typescript, Superbuffer automatically ensures your data has proper typings:

1import {Model, float32} from 'superbuffer';
2const position = Model.fromSchemaDefinition({
3  x: float32,
4  y: float32,
5});
6
7// Typescript knows that only objects with type
8// {x: number, y: number} can be serialized by the `position` model
9const buffer = position.toBuffer({x: 32.1, y: 12.3});
10
11// Typescript knows `result` is of type {x: number, y: number}
12const result = position.fromBuffer(buffer);

Superbuffer can serialize any non-circular JSON structure:

1import {Schema, Model, views, ExtractSchemaObject} from 'superbuffer';
2
3const {int16, int32, uint8, uint32, uint64, float32, boolean, string} = views;
4
5const playerSchema = new Schema({
6  id: uint8,
7  x: float32,
8  y: float32,
9  health: uint8,
10  alive: boolean,
11});
12
13const inputSchema = new Schema({action: int16, movement: int16});
14
15const listSchema = new Schema({value: int32});
16
17const snapshotModel = Model.fromSchemaDefinition({
18  time: uint64,
19  sequenceNumber: uint32,
20  input: inputSchema,
21  messages: [string],
22  data: {
23    list: [listSchema],
24    players: [playerSchema],
25  },
26});
27
28type Snapshot = ExtractSchemaObject<typeof snapshotModel>;
29
30const snapshot: Snapshot = {
31  time: BigInt(Date.now()),
32  sequenceNumber: 438923,
33  input: {
34    action: -12,
35    movement: -4,
36  },
37  messages: ['hello', 'hi', 'how are you', 'im good, how are you', 'fine thank you'],
38  data: {
39    list: [{value: 1}, {value: 2}, {value: 3}, {value: 4}, {value: 5}],
40    players: [
41      {id: 14, x: 145.32, y: 98.1123, health: 99, alive: true},
42      {id: 15, x: 218.46, y: -14.0934, health: 100, alive: true},
43      {id: 0, x: 3289.554, y: -1432.0, health: 0, alive: false},
44    ],
45  },
46};
47
48/** Client */
49// JSON.stringify(snapshot) = 430 bytes
50// model.toBuffer(snapshot) = 139 bytes
51// 68% compression!
52const buffer = snapshotModel.toBuffer(snapshot); // Object to ArrayBuffer
53network.emit(buffer);
54
55/** Server */
56network.on('message', (buffer) => {
57  const id = Model.getIdFromBuffer(buffer);
58  if (id === snapshotModel.schema.id) {
59    const result = snapshotModel.fromBuffer(buffer); // ArrayBuffer to object
60  }
61});

Data Types

Superbuffer's views are mapped to their respective TypedArray countparts:

Superbuffer View	TypedArray Equivalent	Value Range	Byte Size
uint8	Uint8Array	0 to 255	1
uint16	Uint16Array	0 to 65535	2
uint32	Uint32Array	0 to 4294967295	4
uint64	Uint64Array	0 to 2^64-1	8
int8	Int8Array	-128 to 127	1
int16	Int16Array	-32768 to 32767	2
int32	Int32Array	-2147483648 to 2147483647	4
int64	Int64Array	-2^63 to 2^63-1	8
float32	Float32Array	1.2×10^-38 to 3.4×10^38	4
float64	Float64Array	5.0×10^-324 to 1.8×10^308	8
string	Uint8Array	UTF-8 encoding	variable
boolean	Uint8Array	0 or 1	1

Caveats

• Max array length of 65535 (uint16)
• Max number of schema ids is 255 (uint8)
• No " character in strings
• No null/undefined properties: all properties of the schema definition must be present on an object to be serialized (empty arrays are allowed)

License

MIT License