buffer-packer

buffer-packer can pack and unpack an object into/from a structured buffer in a very flexible way.

get started

Assuming you have following C structure (and it is big-endian):

typedef struct __attribute__((__packed__)) {
    uint8_t counter;
    int16_t pos[2];
    char name[12];
    uint8_t dummy[3];  // must be equal 1
    uint8_t sum;       // sum of bytes all above fields
} my_struct_t;

Here is how buffer-packer can be used to create it for you:

const Packer = require("buffer-packer");

const packer = new Packer(
    `counter:u8,
    pos:s16b[2],
    name:data[12],
    :pad[3]=1,
    sum:tap[doSum],
    sum:u8`,
    { doSum: (buf, data) => buf.reduce((acc, cur) => acc + cur, 0) & 0xff }
);

const data = { counter: 7, pos: [0x1234, 0x5678], name: "my test" };

const frame = packer.pack(data);

console.log(frame);
// <Buffer 07 12 34 56 78 6d 79 20 74 65 73 74 00 00 00 00 00 01 01 01 e4>

console.log(packer.unpack(frame));
// {
//   obj: {
//     counter: 7,
//     pos: [ 4660, 22136 ],
//     name: <Buffer 6d 79 20 74 65 73 74 00 00 00 00 00>,
//     sum: 228
//   },
//   baseLength: 21,
//   length: 21
// }

buffer-packer will return any field of type data in a Buffer. If you are sure data is a string simple use .toString() while reading its value:

fields

When you instantiate the Packer object you must supply a format string which is a sequence of fields separated by comma. Each field has as an almost mandatory variable name, a : separator and an always mandatory format specifier.

Apart the format string, you should pass to the class constructor an object with functions in case you are using any tap tag in your fields (see bellow).

integers

You can use use a tag like variable:u16l[2] to include 4 bytes, being 2 unsigned 16 bits values in little endian order from variable[0] and variable[1].

You can also use :s16b=34 to include a constant 34 as a big endian signed 16 bits.

Basic format is:

• optional variable name
• : as separator
• s or u to denote signed and unsigned values
• 8, 16, 32 or 64 to set its size
• l or b to set as little or big endian order (may be absent for size 8 )
• optional [2] or [myLength] to defined field as array and give it a size
• optional =123 to give it a default value in case value not present in input object (mandatory in case variable name not present)

Examples:

• a:u8 value of a as 8 bits unsigned
• b:u32b value of b as 32 bits unsigned in bit-endian order
• c:s16l value of c as 16 bits signed in little-endian order
• d:s8[2] first two elements of array d as 8 bits signed
• e:u8[len] first len elements of array e as 8 bits unsigned
• f:u8=3 value of f as 8 bits unsigned. if absent default to 3
• :s8=-2 a constant 8 bits signed equal to -2

When using a dynamic length as in e:u8[len] the property len MUST be present on input object, event if equal to 0. Also it may be generated by a tap function too.

float

Format is:

• variable name
• : as separator
• should start with f
• 32 or 64 as size
• l or b to set its indianness order.

Examples:

• a:f32b value of a as float 32 bits in bit-endian order

data

This tag can be used for 8 bits data arrays like arrays, Buffer or strings.

Format:

• variable name
• : as separator
• data as type
• [2] or [myLength] to give it a size

Examples:

• a:data[4] first 4 elements of array a. if a.length < 4 it will be padded with 0 to be exact 4
• b:data[len] first len elements of array b. Also receives padding to be exact len bytes long if necessary.

property len MUST be present on input object (event if 0), for the dynamic format

padding

Add some 8 bits padding to align the structure if necessary. Both padding value and length can be defined dynamically.

Format:

• optional variable name
• : as separator
• pad as type
• [2] or [myLength] to give it a size
• optional =1 to change its default value from 0

Examples:

• :pad[3] 3 bytes 0 as padding
• a:pad[2] 2 bytes with value of a or 0 if a absent
• :pad[len] add len bytes 0
• b:pad[1]=255 1 byte with value of b or 255 if b absent

tap

Allow to run code to generate values needed upfront.

Tap function is called with current buffer (as is at the moment tap is being called), and input object.

Function should return a value that will be added to original input object.

To be clear: tap function will NOT modify buffer being generated. It will simple give user a chance to process current buffer to generate some data that will become available to the inserted later

When using a tap function it must be declared as second parameter to Packer constructor.

Format:

• variable name to be ADDED to the object
• : as separator
• tap as type
• [funcName] name of the function to execute

Example:

• sum:tap[doSum] run function doSum and add its return value as property sum on original input object

On bellow example note that the value of sum being added by the last field isn't available on original input data until tap function is called.

Also note buffer argument inside tap function has the length of all data processed up to that moment.

function doSum(buffer, data) {
    console.log(buffer, data);
    // outputs: <Buffer 01 02> {id: 1, func: 2}
    acc = 0;
    buffer.forEach((v) => (acc += v));
    return acc;
}

const packer = new Packer(
    `id:u8,
    func:u8,
    sum:tap[doSum],
    sum:u16b`,
    { doSum }
);

console.log(packer.pack({ id: 1, func: 2 }));
// outputs: <Buffer 01 02 00 03>

unpacking

Once you have a packer instance you can feed it with a Buffer to get the original data object used to pack it. Example:

const packer = new Packer(
    `a:u8,
    b:u32l,
    :pad[3],
    text:data[2],
    c:u8[len],
    z:u8=12,
    :u8=14`
);

const data = {
    a: 5,
    b: 0x12345678,
    c: [1, 2, 3, 4],
    text: "string",
    len: 3,
};
const frame = packer.pack(data);

console.log(frame);
// <Buffer 05 78 56 34 12 00 00 00 73 74 01 02 03 0c 0e>

console.log(packer.unpack(frame.slice(0, 5), { len: 3 }));
// {err: 'too short'}

console.log(packer.unpack(frame));
// throw "Missing dynamic size `len`"

console.log(packer.unpack(frame, { len: 3 }));
// {
//   obj: {
//     len: 3,
//     a: 5,
//     b: 305419896,
//     text: <Buffer 73 74>,
//     c: [ 1, 2, 3 ],
//     z: 12
//   },
//   baseLength: 12,
//   length: 15
// }

Important points to note:

• note we had to provide a starting data object to unpack with len initialized so it knows the dynamic size was used to pack c property
• data field always returns as Buffer (see the text field above). Simple use .toString() to get original string if needed
• the unpacked c and text are both smaller then original values since we have only packed part of them
• the last field does not appears on the result since it's an unnamed constant value
• z field appear event not being present on original data object. This happens because it has a default value declared
• the returned object has the property obj when the unpacking succeed or err when it fails (in this case err is a string with a brief description of the cause of failure)
• when unpacking succeed you also get a baseLength and length properties. Note that if you have any dynamically sized property in your pack then your length will change between packs. (baseLength is the minimum length assuming all dynamic length are set to 0)
• recoverable errors like being too short or wrong default do not throw, so you can wait for more data an try again. In other hand an error like Missing dynamic size will throw since is likely to be a design error.

Parsing works one tag per turn, and parsed values are appended to result object as soon they are available. In previous example we had to supply the value of len from beginning, but, if we had a tag resolving the value of len before it was needed (to define c´s size in that example), then we could execute unpack without any starting object.