pbf

A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding protocol buffers, a compact binary format for structured data serialization. Works both in Node and the browser. Supports lazy decoding and detailed customization of the reading/writing code.

Performance

This library is extremely fast — much faster than native JSON.parse/JSON.stringify and the protocol-buffers module. Here's a result from running a real-world benchmark on Node v6.5 (decoding and encoding a sample of 439 vector tiles, 22.6 MB total):

• pbf decode: 387ms, or 57 MB/s
• pbf encode: 396ms, or 56 MB/s
• protocol-buffers decode: 837ms, or 26 MB/s
• protocol-buffers encode: 4197ms, or 5 MB/s
• JSON.parse: 1540ms, or 125 MB/s (parsing an equivalent 77.5 MB JSON file)
• JSON.stringify: 607ms, or 49 MB/s

Examples

Using Compiled Code

Install pbf and compile a JavaScript module from a .proto file:

1$ npm install -g pbf
2$ pbf example.proto > example.js

Then read and write objects using the module like this:

1import Pbf from 'pbf';
2import {readExample, writeExample} from './example.js';
3
4// read
5var obj = readExample(new Pbf(buffer));
6
7// write
8const pbf = new Pbf();
9writeExample(obj, pbf);
10const buffer = pbf.finish();

Alternatively, you can compile a protobuf schema file directly in the code:

1import {compile} from 'pbf/compile';
2import schema from 'protocol-buffers-schema';
3
4const proto = schema.parse(fs.readFileSync('example.proto'));
5const {readExample, writeExample} = compile(proto);

Custom Reading

1var data = new Pbf(buffer).readFields(readData, {});
2
3function readData(tag, data, pbf) {
4    if (tag === 1) data.name = pbf.readString();
5    else if (tag === 2) data.version = pbf.readVarint();
6    else if (tag === 3) data.layer = pbf.readMessage(readLayer, {});
7}
8function readLayer(tag, layer, pbf) {
9    if (tag === 1) layer.name = pbf.readString();
10    else if (tag === 3) layer.size = pbf.readVarint();
11}

Custom Writing

1var pbf = new Pbf();
2writeData(data, pbf);
3var buffer = pbf.finish();
4
5function writeData(data, pbf) {
6    pbf.writeStringField(1, data.name);
7    pbf.writeVarintField(2, data.version);
8    pbf.writeMessage(3, writeLayer, data.layer);
9}
10function writeLayer(layer, pbf) {
11    pbf.writeStringField(1, layer.name);
12    pbf.writeVarintField(2, layer.size);
13}

Install

Install using NPM with npm install pbf, then import as a module:

1import Pbf from 'pbf';

Or use as a module directly in the browser with jsDelivr:

1<script type="module">
2    import Pbf from 'https://cdn.jsdelivr.net/npm/pbf/+esm';
3</script>

Alternatively, there's a browser bundle with a Pbf global variable:

1<script src="https://cdn.jsdelivr.net/npm/pbf"></script>

API

Create a Pbf object, optionally given a Buffer or Uint8Array as input data:

1// parse a pbf file from disk in Node
2const pbf = new Pbf(fs.readFileSync('data.pbf'));
3
4// parse a pbf file in a browser after an ajax request with responseType="arraybuffer"
5const pbf = new Pbf(new Uint8Array(xhr.response));

Pbf object properties:

1pbf.length; // length of the underlying buffer
2pbf.pos; // current offset for reading or writing

Reading

Read a sequence of fields:

1pbf.readFields((tag) => {
2    if (tag === 1) pbf.readVarint();
3    else if (tag === 2) pbf.readString();
4    else ...
5});

It optionally accepts an object that will be passed to the reading function for easier construction of decoded data, and also passes the Pbf object as a third argument:

1const result = pbf.readFields(readField, {})
2
3function readField(tag, result, pbf) {
4    if (tag === 1) result.id = pbf.readVarint();
5}

To read an embedded message, use pbf.readMessage(fn[, obj]) (in the same way as read).

Read values:

1const value = pbf.readVarint();
2const str = pbf.readString();
3const numbers = pbf.readPackedVarint();

For lazy or partial decoding, simply save the position instead of reading a value, then later set it back to the saved value and read:

1const fooPos = -1;
2pbf.readFields((tag) => {
3    if (tag === 1) fooPos = pbf.pos;
4});
5...
6pbf.pos = fooPos;
7pbf.readMessage(readFoo);

Scalar reading methods:

• readVarint(isSigned) (pass true if you expect negative varints)
• readSVarint()
• readFixed32()
• readFixed64()
• readSFixed32()
• readSFixed64()
• readBoolean()
• readFloat()
• readDouble()
• readString()
• readBytes()
• skip(value)

Packed reading methods:

• readPackedVarint(arr, isSigned) (appends read items to arr)
• readPackedSVarint(arr)
• readPackedFixed32(arr)
• readPackedFixed64(arr)
• readPackedSFixed32(arr)
• readPackedSFixed64(arr)
• readPackedBoolean(arr)
• readPackedFloat(arr)
• readPackedDouble(arr)

Writing

Write values:

1pbf.writeVarint(123);
2pbf.writeString("Hello world");

Write an embedded message:

1pbf.writeMessage(1, writeObj, obj);
2
3function writeObj(obj, pbf) {
4    pbf.writeStringField(obj.name);
5    pbf.writeVarintField(obj.version);
6}

Field writing methods:

• writeVarintField(tag, val)
• writeSVarintField(tag, val)
• writeFixed32Field(tag, val)
• writeFixed64Field(tag, val)
• writeSFixed32Field(tag, val)
• writeSFixed64Field(tag, val)
• writeBooleanField(tag, val)
• writeFloatField(tag, val)
• writeDoubleField(tag, val)
• writeStringField(tag, val)
• writeBytesField(tag, buffer)

Packed field writing methods:

• writePackedVarint(tag, val)
• writePackedSVarint(tag, val)
• writePackedSFixed32(tag, val)
• writePackedSFixed64(tag, val)
• writePackedBoolean(tag, val)
• writePackedFloat(tag, val)
• writePackedDouble(tag, val)

Scalar writing methods:

• writeVarint(val)
• writeSVarint(val)
• writeSFixed32(val)
• writeSFixed64(val)
• writeBoolean(val)
• writeFloat(val)
• writeDouble(val)
• writeString(val)
• writeBytes(buffer)

Message writing methods:

• writeMessage(tag, fn[, obj])
• writeRawMessage(fn[, obj])

Misc methods:

• realloc(minBytes) - pad the underlying buffer size to accommodate the given number of bytes; note that the size increases exponentially, so it won't necessarily equal the size of data written
• finish() - make the current buffer ready for reading and return the data as a buffer slice

For an example of a real-world usage of the library, see vector-tile-js.

Proto Schema to JavaScript

If installed globally, pbf provides a binary that compiles proto files into JavaScript modules. Usage:

1$ pbf <proto_path> [--no-write] [--no-read] [--legacy]

The --no-write and --no-read switches remove corresponding code in the output. The --legacy switch makes it generate a CommonJS module instead of ESM.

Pbf will generate read<Identifier> and write<Identifier> functions for every message in the schema. For nested messages, their names will be concatenated — e.g. Message inside Test will produce readTestMessage and writeTestMessage functions.

• read(pbf) - decodes an object from the given Pbf instance.
• write(obj, pbf) - encodes an object into the given Pbf instance (usually empty).

The resulting code is clean and simple, so it's meant to be customized.