npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@mapwhit/pbf

v1.0.2

Published

a low-level, lightweight protocol buffers implementation in JavaScript

Downloads

26

Readme

NPM version Build Status Dependency Status

@mapwhit/pbf

Fork of pbf

A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding protocol buffers, a compact binary format for structured data serialization. Works both in Node and the browser. Supports lazy decoding and detailed customization of the reading/writing code.

Performance

This library is extremely fast — much faster than native JSON.parse/JSON.stringify and the protocol-buffers module. Here's a result from running a real-world benchmark on Node v6.5 (decoding and encoding a sample of 439 vector tiles, 22.6 MB total):

  • pbf decode: 387ms, or 57 MB/s
  • pbf encode: 396ms, or 56 MB/s
  • protocol-buffers decode: 837ms, or 26 MB/s
  • protocol-buffers encode: 4197ms, or 5 MB/s
  • JSON.parse: 1540ms, or 125 MB/s (parsing an equivalent 77.5 MB JSON file)
  • JSON.stringify: 607ms, or 49 MB/s

Examples

Using Compiled Code

Install pbf and compile a JavaScript module from a .proto file:

$ npm install -g pbf
$ pbf example.proto > example.js

Then read and write objects using the module like this:

var Pbf = require('pbf');
var Example = require('./example.js').Example;

// read
var pbf = new Pbf(buffer);
var obj = Example.read(pbf);

// write
var pbf = new Pbf();
Example.write(obj, pbf);
var buffer = pbf.finish();

Alternatively, you can compile a module directly in the code:

var compile = require('pbf/compile');
var schema = require('protocol-buffers-schema');

var proto = schema.parse(fs.readFileSync('example.proto'));
var Test = compile(proto).Test;

If you use webpack as your module bundler, you can use pbf-loader to load .proto files directly. It returns a compiled module ready to be used.

Given you already configured your webpack.config.js, the code above would look like:

var Pbf = require('pbf');
var proto = require('./example.proto');

var Test = proto.Test;

Custom Reading

var data = new Pbf(buffer).readFields(readData, {});

function readData(tag, data, pbf) {
    if (tag === 1) data.name = pbf.readString();
    else if (tag === 2) data.version = pbf.readVarint();
    else if (tag === 3) data.layer = pbf.readMessage(readLayer, {});
}
function readLayer(tag, layer, pbf) {
    if (tag === 1) layer.name = pbf.readString();
    else if (tag === 3) layer.size = pbf.readVarint();
}

Custom Writing

var pbf = new Pbf();
writeData(data, pbf);
var buffer = pbf.finish();

function writeData(data, pbf) {
    pbf.writeStringField(1, data.name);
    pbf.writeVarintField(2, data.version);
    pbf.writeMessage(3, writeLayer, data.layer);
}
function writeLayer(layer, pbf) {
    pbf.writeStringField(1, layer.name);
    pbf.writeVarintField(2, layer.size);
}

Install

Node and Browserify:

npm install pbf

Making a browser build:

npm install
npm run build-dev # dist/pbf-dev.js (development build)
npm run build-min # dist/pbf.js (minified production build)

CDN link: https://unpkg.com/[email protected]/dist/pbf.js

API

Create a Pbf object, optionally given a Buffer or Uint8Array as input data:

// parse a pbf file from disk in Node
var pbf = new Pbf(fs.readFileSync('data.pbf'));

// parse a pbf file in a browser after an ajax request with responseType="arraybuffer"
var pbf = new Pbf(new Uint8Array(xhr.response));

Pbf object properties:

pbf.length; // length of the underlying buffer
pbf.pos; // current offset for reading or writing

Reading

Read a sequence of fields:

pbf.readFields(function (tag) {
    if (tag === 1) pbf.readVarint();
    else if (tag === 2) pbf.readString();
    else ...
});

It optionally accepts an object that will be passed to the reading function for easier construction of decoded data, and also passes the Pbf object as a third argument:

var result = pbf.readFields(callback, {})

function callback(tag, result, pbf) {
    if (tag === 1) result.id = pbf.readVarint();
}

To read an embedded message, use pbf.readMessage(fn[, obj]) (in the same way as read).

Read values:

var value = pbf.readVarint();
var str = pbf.readString();
var numbers = pbf.readPackedVarint();

For lazy or partial decoding, simply save the position instead of reading a value, then later set it back to the saved value and read:

var fooPos = -1;
pbf.readFields(function (tag) {
    if (tag === 1) fooPos = pbf.pos;
});
...
pbf.pos = fooPos;
pbf.readMessage(readFoo);

Scalar reading methods:

  • readVarint(isSigned) (pass true if you expect negative varints)
  • readSVarint()
  • readFixed32()
  • readFixed64()
  • readSFixed32()
  • readSFixed64()
  • readBoolean()
  • readFloat()
  • readDouble()
  • readString()
  • readBytes()
  • skip(value)

Packed reading methods:

  • readPackedVarint(arr, isSigned) (appends read items to arr)
  • readPackedSVarint(arr)
  • readPackedFixed32(arr)
  • readPackedFixed64(arr)
  • readPackedSFixed32(arr)
  • readPackedSFixed64(arr)
  • readPackedBoolean(arr)
  • readPackedFloat(arr)
  • readPackedDouble(arr)

Writing

Write values:

pbf.writeVarint(123);
pbf.writeString("Hello world");

Write an embedded message:

pbf.writeMessage(1, writeObj, obj);

function writeObj(obj, pbf) {
    pbf.writeStringField(obj.name);
    pbf.writeVarintField(obj.version);
}

Field writing methods:

  • writeVarintField(tag, val)
  • writeSVarintField(tag, val)
  • writeFixed32Field(tag, val)
  • writeFixed64Field(tag, val)
  • writeSFixed32Field(tag, val)
  • writeSFixed64Field(tag, val)
  • writeBooleanField(tag, val)
  • writeFloatField(tag, val)
  • writeDoubleField(tag, val)
  • writeStringField(tag, val)
  • writeBytesField(tag, buffer)

Packed field writing methods:

  • writePackedVarint(tag, val)
  • writePackedSVarint(tag, val)
  • writePackedSFixed32(tag, val)
  • writePackedSFixed64(tag, val)
  • writePackedBoolean(tag, val)
  • writePackedFloat(tag, val)
  • writePackedDouble(tag, val)

Scalar writing methods:

  • writeVarint(val)
  • writeSVarint(val)
  • writeSFixed32(val)
  • writeSFixed64(val)
  • writeBoolean(val)
  • writeFloat(val)
  • writeDouble(val)
  • writeString(val)
  • writeBytes(buffer)

Message writing methods:

  • writeMessage(tag, fn[, obj])
  • writeRawMessage(fn[, obj])

Misc methods:

  • realloc(minBytes) - pad the underlying buffer size to accommodate the given number of bytes; note that the size increases exponentially, so it won't necessarily equal the size of data written
  • finish() - make the current buffer ready for reading and return the data as a buffer slice
  • destroy() - dispose the buffer

For an example of a real-world usage of the library, see vector-tile-js.

Proto Schema to JavaScript

If installed globally, pbf provides a binary that compiles proto files into JavaScript modules. Usage:

$ pbf <proto_path> [--no-write] [--no-read] [--browser]

The --no-write and --no-read switches remove corresponding code in the output. The --browser switch makes the module work in browsers instead of Node.

The resulting module exports each message by name with the following methods:

  • read(pbf) - decodes an object from the given Pbf instance
  • write(obj, pbf) - encodes an object into the given Pbf instance (usually empty)

The resulting code is clean and simple, so feel free to customize it.