binary-packet

Lightweight and hyper-fast, zero-dependencies, TypeScript-first, schema-based binary packets serialization and deserialization library.
Originally made to be used for an ultrafast WebSockets communication with user-defined type-safe messages between client and server, with the smallest bytes usage possible.

Supports serializing into and deserializing from DataViews, ArrayBuffers and Buffers (NodeJS/Bun only).
To achieve the maximum performance is it always advised to use node Buffer(s) when available.

Installation

Node:
npm install binary-packet

Bun:
bun add binary-packet

Features & Specification

Define the structure of the packets through unique Packet IDs and "schema" objects.
This "schema" object is simply called Definition and defines the shape of a packet: specifically its fields and their types.

Fields / Data types

Currently, these kinds of fields are supported: | Type | Description | Values | Size (bytes) | |------|-------------|--------------|--------------| | Field.UNSIGNED_INT_8 | 8 bits unsigned integer | 0 - 255 | 1 | | Field.UNSIGNED_INT_16 | 16 bits unsigned integer | 0 - 65535 | 2 | | Field.UNSIGNED_INT_32 | 32 bits unsigned integer | 0 - 4294967295 | 4 | | Field.INT_8 | 8 bits signed integer | -128 - 127 | 1 | | Field.INT_16 | 16 bits signed integer | -32768 - 32767 | 2 | | Field.INT_32 | 32 bits signed integer | -2147483648 - 2147483647 | 4 | | Field.FLOAT_32 | 32 bits IEEE754 floating-point | | 4 | | Field.FLOAT_64 | 64 bits IEEE754 floating-point | | 8 | | BinaryPacket | BinaryPacket "subpacket" | BinaryPacket | size(BinaryPacket) | | FieldString | ASCII or single octet utf-8 chars string | Up to 65536 chars | 2 + length | | FieldArray | Dynamically-sized array of one of the types above | Up to 256 elements | 1 + length * size(Element) | | FieldFixedArray | Statically-sized array of one of the types above | Any pre-defined numbers of elements | length * size(Element) | | FieldBitFlags | Boolean flags packed into a single 8 bits integer | Up to 8 boolean flags | 1 | | FieldOptional | Optional BinaryPacket "subpacket" | BinaryPacket | undefined | 1 + size(BinaryPacket) |

As shown, both arrays and nested objects ("subpackets") are supported.
Note: FieldFixedArray is much more memory efficient and performant than FieldArray, but require a pre-defined length.

Pattern matching

The library exposes an easy way to "pattern match" packets of a yet-unknown-type in a type-safe manner through a visitor pattern.
For an example, search for "pattern matching" in the examples below.

Usage Examples

Example: (incomplete) definition of a simplistic board game

import { BinaryPacket, Field, FieldArray } from 'binary-packet'

// Suppose we have a game board where each cell is a square and is one unit big.
// A cell can be then defined by its X and Y coordinates.
// For simplicity, let's say there cannot be more than 256 cells, so we can use 8 bits for each coordinate.
const Cell = {
  x: Field.UNSIGNED_INT_8,
  y: Field.UNSIGNED_INT_8
}

// When done with the cell definition we can create its BinaryPacket writer/reader.
// NOTE: each BinaryPacket needs an unique ID, for identification purposes and error checking.
const CellPacket = BinaryPacket.define(0, Cell)

// Let's now make the definition of the whole game board.
// You can also specify arrays of both "primitive" fields and other BinaryPackets.
const Board = {
  numPlayers: Field.UNSIGNED_INT_8,
  otherStuff: Field.INT_32,
  cells: FieldArray(CellPacket)
}

// When done with the board definition we can create its BinaryPacket writer/reader.
// NOTE: each BinaryPacket needs an unique ID, for identification purposes and error checking.
const BoardPacket = BinaryPacket.define(1, Board)

//////////////////
// WRITING SIDE //
//////////////////
const buffer = BoardPacket.writeNodeBuffer({
  numPlayers: 1,
  otherStuff: 69420,
  cells: [
    { x: 0, y: 0 },
    { x: 1, y: 1 }
  ]
})

// ...
// sendTheBufferOverTheNetwork(buffer)
// ...

//////////////////
// READING SIDE //
//////////////////
import assert from 'assert'

// ...
// const buffer = receiveTheBufferFromTheNetwork()
// ...

const board = BoardPacket.readNodeBuffer(buffer)

assert(board.numPlayers === 1)
assert(board.otherStuff === 69420)
assert(board.cells.length === 2)
assert(board.cells[0].x === 0)
assert(board.cells[0].y === 0)
assert(board.cells[1].x === 1)
assert(board.cells[1].y === 1)

Example: pattern matching

import assert from 'assert/strict'
import { BinaryPacket, Field } from 'binary-packet'

// Packet A definition
const A = BinaryPacket.define(1)

// Packet B definition: This is the kind of packets that we care about in this example!
const B = BinaryPacket.define(2, { data: Field.UNSIGNED_INT_8 })

// Packet C definition
const C = BinaryPacket.define(3)

// Assume the following packet comes from the network or, for some other reason, is a buffer we do not know anything about.
const buffer = B.writeNodeBuffer({ data: 255 })

BinaryPacket.visitNodeBuffer(
  buffer,

  A.visitor(() => assert(false, 'Erroneously accepted visitor A')),

  B.visitor(packet => {
    // Do something with the packet
    assert.equal(packet.data, 255)
    console.log('Accepted visitor B:', packet)
  }),

  C.visitor(() => assert(false, 'Erroneously accepted visitor C'))
)

Benchmarks & Alternatives

Benchmarks are not always meant to be taken seriously.
Most of the times the results of a benchmark do not actually show the full capabilities of each library.
So, take these "performance" comparisons with a grain of salt; or, even better, do your own benchmarks with the actual data you need to serialize/deserialize.

This library has been benchmarked against the following alternatives:

• msgpackr - A very popular, fast and battle-tested library. Currently offers more features than binary-packet, but it always appears to be slower and is also less type-safe.
• restructure - An older, popular schema-based library, has some extra features like LazyArrays, but it is much slower than both binary-packet and msgpackr. And, sadly, easily crashes with complex structures.

The benchmarks are executed on three different kinds of packets:

• EmptyPacket: basically an empty javascript object.
• SimplePacket: objects with just primitive fields, statically-sized arrays and a string.
• ComplexPacket: objects with primitives, statically-sized arrays, dynamically-sized arrays, bitflags, a string, an array of strings and other nested objects/arrays.

You can see and run the benchmarks yourself if you:

• Clone the repository.
• Launch npm run benchmark.

Disclaimer

This library is still very new, thus not "battle-tested" in production enough, or may still have missing important features.
If you plan on serializing highly sensitive data or need to guarantee no crashes, use an alternative like msgpackr until this library becomes 100% production-ready.

Contribute

Would like to have more complex, but still hyper-fast and memory efficient, features?
Contribute on GitHub yourself or, alternatively, buy me a coffee!