Encode Tools

This package aggregates different libraries for encoding, serializing, compressing, generating ids and hashing things, exposing a common interface.

Many other packages serve the same purpose, but our objective is to ensure a consistent experience in both node.js and the browser and standardize the api so functions work the same way across different underlying libraries.

Encode Tools also has a command line wrapper encode-cli.

Examples

Encoding a Buffer as base64url

  let enc = new EncodeTools();
  let buf = Buffer.from('hello world', 'utf8');
  let newBuf = enc.encodeBuffer(buf, BinaryEncoding.base64url);
  console.log(newBuf.toString('utf8'));

Hashing an object wth xxhash

let enc = new EncodeTools();
let obj = { foo: 'bar' };
let newBuf = await enc.hashObject(obj, HashAlgorithm.xxhash64);
console.log(newBuf.toString('utf8'));

Serializing an object wth msgpack

let enc = new EncodeTools();
let obj = { foo: 'bar' };
let newBuf = await enc.serializeObject(obj, SerializationFormat.msgpack);
console.log(newBuf.toString('base64'));

Generating a base64-encoded UUID v4

let enc = new EncodeTools();
let newBuf = await enc.uniqueId(IDFormat.uuidv4);
console.log(newBuf.toString('base64'));

Compressing a buffer with lzma

let enc = new EncodeTools();
let newBuf = await enc.compress(Buffer.from('hi', 'utf8'), CompressionFormat.lzma);
console.log(newBuf.toString('base64'));

Resizing a png image

let enc = new EncodeTools();
let imageBuf = await (await new Promise((resolve, reject) => {
  new (Jimp)(500, 500, '#FFFFFF', (err: unknown, image: any) => {
    if (err) reject(err);
    else resolve(image);
  });
})).getBufferAsync('image/png');

let myResizedPng = await enc.resizeImage(imageBuf, { width: 250 }, ImageFormat.png);

Structure

This project is divided into two packages, a core package with no native dependencies (@znetstar/encode-tools), and a full version with native modules as optional dependencies (@znetstar/encode-tools-native). The core package should work well in the browser via Webpack or Browserify, whereas the full version includes more formats and should have better performance.

Algorithms

Below are a list of supported algorithms, their backing library, and their support in the browser.

Binary Encoding

| Name | Works In Browser? | Package | Native Package | | ---------------- | ----------------- | ---------------------------------------------------- | ------------------------------------------------ | | nodeBuffer | ✓ | buffer | (built-in) | | base64 | ✓ | buffer | (built-in) | | base64url | ✓ | buffer | (built-in) | | hex | ✓ | buffer | | | base32 | ✓ | base32.js | | | hashids | ✓ | hashids | | | arrayBuffer | ✓ | (built-in) | | | base85 (ascii85) | ✓ | base85 | | | ascii85 | ✓ | base85 | |

Hashing

| Name | Works In Browser? | Package | Native Package | | ------------- | ----------------- | ---------------------------------------------------- | ---------------------------------------------------------- | | crc32 | ✓ | hash-wasm | | | xxhash3 | ✓ | hash-wasm | xxhash-addon | | xxhash64 | ✓ | hash-wasm | xxhash-addon | | xxhash32 | ✓ | hash-wasm | xxhash-addon | | md5 | ✓ | hash-wasm | (built-in) | | sha1 | ✓ | hash-wasm | (built-in) | | sha2 (sha512) | ✓ | hash-wasm | (built-in) | | sha512 | ✓ | hash-wasm | (built-in) | | sha3 | ✓ | hash-wasm | | | bcrypt | ✓ | hash-wasm | |

ID Generation

| Name | Works In Browser? | Package | Native Package | | ------------ | ----------------- | ---------- | -------------- | | uuidv4 | ✓ | uuid | | | uuidv1 | ✓ | uuid | | | uuidv4string | ✓ | uuid | | | uuidv1string | ✓ | uuid | | | objectId | ✓ | bson | bson-ext | | nanoid | ✓ | nanoid | | | timestamp | ✓ | (built in) | |

Serialization

| Name | Works In Browser? | Package | Native Package | |---------|-------------------|------------------|----------------| | json | ✓ | (built in) | | | cbor | ✓ | cbor-web | cbor | | msgpack | ✓ | @msgpack/msgpack | | | bson | ✓ | bson | bson-ext | | json5 | ✓ | json5 | |

Compression

| Name | Works In Browser? | Package | Native Package | | ---- | ----------------- | ---------- | -------------- | | zstd | ✓ | zstd-codec | | | lzma | ✓ | lzma | lzma-native |

Image Manipulation

| Name | Works In Browser? | Package | Native Package | | ---- | ----------------- | ------- | -------------- | | png | ✓ | jimp | sharp | | jpeg | ✓ | jimp | sharp | | webp | | sharp | | | avif | | sharp | | | tiff | | sharp | | | gif* | | sharp | |

Requirements

Encode Tools runs in the browser and in node.js, with a few exceptions. The bson-ext, lzma-native, xxhash-addon and cbor-extract packages have native bindings, and so cannot run in the browser. For browser compatibility, the EncodeTools class uses the pure javascript bson, lzma, hash-wsam, and cbor-x packages, respectively, to provide equivalent support albeit at the cost of performance. Additionally, hash-wsam lacks support for xxhash3.

The EncodeToolsAuto class will use the native packages bson-ext, lzma-native and xxhash-addon (and any future native packages). bson-ext, lzma-native and xxhash-addon are listed as optional dependencies, and NPM will attempt to install them automatically.

The constructor of EncodeToolsAuto takes a second set of default EncodingOptions to use as a fallback if it cannot find the needed module.

const enc = new EncodeToolsAuto({ hashAlgorithm: HashAlgorithm.xxhash3 }, { hashAlgorithm: HashAlgorithm.xxhash64 });
if (enc.availableNativeModules.xxhashAddon)
    console.log('should be xxhash3', await enc.hashString('Test'));
else
    console.log('should be xxhash64', await enc.hashString('Test'));

The gif image format in EncodeToolsAuto requires libvips compiled with ImageMagick support (as described here). I haven't had time to re-build libvips on my machine, so there are no mocha tests for the gif format.

Usage

Please see the documentation located at https://znetstar.github.io/encode-tools/ and https://znetstar.github.io/encode-tools-native/

Webpack

For issues with Webpack, try adding all the native dependencies to the externals section.

{
  externals: {
      'xxhash-addon': 'commonjs xxhash-addon',
      'bson-ext': 'commonjs bson-ext',
      'shelljs': 'commonjs shelljs',
      'lzma-native': 'commonjs lzma-native',
      'sharp': 'commonjs sharp'
  },
  // For Webpack 5+ only, add `node-polyfill-webpack-plugin`
  plugins: [
    new (require('node-polyfill-webpack-plugin'))()
  ]
}

Next.js

For Next.js, you can insert into next.config.js

{
  webpack: (config, { isServer }) => {
    if (!isServer) {
      config.resolve.fallback = {
        fs: false
      }
      config.externals = {
        ...config.externals,
        'xxhash-addon': 'commonjs xxhash-addon',
        'bson-ext': 'commonjs bson-ext',
        'shelljs': 'commonjs shelljs',
        'lzma-native': 'commonjs lzma-native',
        'sharp': 'commonjs sharp'
      }
      config.plugins = [
        ...config.plugins,
        new (require('node-polyfill-webpack-plugin'))()
      ]
    }
    return config;
  }
}

Tests

Tests are written in Mocha, to run use npm test.

Attribution

This project is a fork of @etomon/encode-tools.

License

znetstar Encode Tools is licensed under the GNU LGPL-3.0, a copy of which can be found at https://www.gnu.org/licenses/.