npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

bidar

v0.4.5

Published

Binary Data Representation (serialization format)

Downloads

50

Readme

Bidar: Binary Data Representation.

Oh no! Not another BSON?!?!

Actually, yes.

The fact of the proliferation of serialized data formats, and of binary JSON-like formats in particular, is a fiat demonstration that there isn't actually a solid consensus for a single unified and universal format. As such, I (Danfuzz) think it's a reasonable and prudent idea to continue the exploration of the design of these things.

To be clear, I am not about to claim that the format defined here is the One True Serialized JavaScript-Like Object Format. What I will claim, though, is that this one is at a significantly different spot in the coordinate-space of possible serialization formats compared to the other ones that showed up on my radar. As such, this one may turn out to be useful to folks in cases where the other existing solutions don't seem to be particularly apropos.

Here are the properties being aimed for here:

  • reasonably space-efficient

  • single-representation (non-equal binary forms implies non-isomorphic / non-equal object forms)

  • without gratuitous quoting (e.g., look what happens when you put a JSON-encoded string inside another JSON-encoded string; avoid that!)

  • supports a buffer (binary blob) type

  • embraces as much of the JavaScript data model as is reasonably possible. For example, this format differentiates null and undefined. Why? Because you can tell the difference between them in the purely-local case, and you shouldn't necessarily have to change your code if you decide to interpose a serialization (e.g., a machine boundary). I have no love of undefined myself and wish I could do away with it, but I'm not about to try to do so wishfully.

  • supports "holes" where non-pure-data elements may be filled in. More specifically, supports clients that want to be able to set up function/callback-like things for over-the-wire communication.

  • supports non-tree graphs of objects. This includes both circular structures and arbitrary acyclic graphs. Non-cycles might be tempting to convert to trees on serialization, but that would have poor interplay with the idea of "holes" (above). That is, once you have "holes," object identity starts to matter a lot more.

Non-Goals:

  • partially streamable. If you want to incrementally stream a (notionally) single structure, that can be done by defining a layer above this one.

  • random-access. If you need an index into a structure, then (as with the previous item) it can be built as an additional layer.

A few references:

Example

One of the major reasons for the development of this module was to have the ability to sensibly manage and manipulate non-data "holes" in what is otherwise a graph of data objects, particularly so that two sides of a communication link can succeed in communicating about at least some "non-data-y things" despite not being able to actually share an address space.

In this example, imagine that the two sides (serializing and parsing) do a bit more than the trivial transformations here. Hopefully, this is enough to give a flavor of things.

(You can find this example in the example directory.)

var bidar = require("../");

function muffins() {
    return "muffins";
}

function are() {
    return "are";
}

function tasty() {
    return "tasty";
}

function holeFilter(hole) {
    return "[" + hole() + "]";
}

function holeReverser(hole) {
    return "<" + hole + ">";
}

var message = [ muffins, are, "very", tasty ];

// The "sending" side.
var serialized = bidar.serialize(message, holeFilter);

// The "receiving" side.
var parsed = bidar.parse(serialized, holeReverser);
console.log(parsed);

Here is a transcript of running it:

$ node example/transform.js
[ '<[muffins]>', '<[are]>', 'very', '<[tasty]>' ]

Format Details

The format will be more fully described once it has baked a bit and seems stable. In the mean time, refer to the source. (It's got comments.)

Usage

This library provides just a few top-level functions.

serialize(root, [holeFilter])

Serialize an object graph, rooted at the given object, into binary form. Returns a buffer of the result. The hole filter, if provided is used as a transformation on "holes" encoutered in the object graph.

A "hole" is any object that is not representable as pure data. This includes functions and (non-array, non-buffer) objects whose prototype is not the default object prototype.

The hole filter is called with a single argument -- the original "hole" object encountered in the graph -- and is expected to return a serializable pure data object in response. That object is marked as a "hole replacement" in the serialized output.

function parse(buf, [holeFiller])

Parse a serialized form into an object graph, returning the root of the graph as originally serialized. The holeFiller, if specified, is a function which is consulted to transform "hole replacements" (see the description of serialize() for more details) back into directly-usable objects.

The entire contents of the given buffer are expeceted to be consulted in the process of parsing. If there is extra data at the end of the buffer that is not needed, then this function throws an exception.

This will also throw an exception if there is insufficient data to complete the operation.

And finally, this will throw an exception if given a serialized form that has holes, if holeFiller was not specified.

parsePartial(buf, [startIndex], [holeFiller])

This is like parse(), except that only a portion of the input buffer is expected to be used. The startIndex (defaults to 0) is the index into the buffer at which to begin parsing. The optional holeFilter is the same as with parse().

The return value of this call is a map of two values: root is bound to the root of the parsed object graph. bytesConsumed is bound to the number of bytes that were read from the buffer during the parse operation.

type

This is a sub-module that contains convenient utilities for type determination and related assertions. The author does not particularly recommend the use of this module. It may drastically change over time, or may simply disappear.

Building and Installing

npm install bidar

Or grab the source and

node-waf configure build

Testing

npm test

Or

node ./test/test.js

To Do

Known Deficiencies:

  • If there are extra properties on Buffer objects, they get silently ignored. E.g.: On serialization, foo will get dropped with this definition of x: x = new Buffer(10); x.foo = "bar";

  • Regex objects are currently considered holes. They should perhaps be handled specially (as Buffer objects currently are).

  • Date objects are currently considered holes. They should perhaps be handled specially (as Buffer objects currently are).

Contributing

Questions, comments, bug reports, and pull requests are all welcome. Submit them at the project on GitHub.

Bug reports that include steps-to-reproduce (including code) are the best. Even better, make them in the form of pull requests that update the test suite. Thanks!

Author

Dan Bornstein (personal website), supported by The Obvious Corporation.

License

Copyright 2012 The Obvious Corporation.

Licensed under the Apache License, Version 2.0. See the top-level file LICENSE.txt and (http://www.apache.org/licenses/LICENSE-2.0).