Wasm Chunking Fastcdc

This crate generates WebAssembly bindings to the FastCDC content defined slicing library.

Install for nodejs usage

npm install @dstanesc/wasm-chunking-fastcdc-node

Install for bundler usage

npm install @dstanesc/wasm-chunking-fastcdc-webpack

Usage TypeScript

//import { compute_chunks } from "@dstanesc/wasm-chunking-fastcdc-node"
import { compute_chunks } from "@dstanesc/wasm-chunking-fastcdc-webpack"
const buf: Uint8Array  = ...
// chunking  spec: min_size 16 KB, avg size 32 KB, max size 64 KB
const offsets: Uint32Array = compute_chunks(buf, 16384, 32768, 65536)   

Wasm build requirements

• Rust toolchain
• Wasm-pack

Wasm build

wasm-pack build --out-dir pkg/webpack --out-name chunking --target bundler --scope dstanesc
wasm-pack build --out-dir pkg/node --out-name chunking --target nodejs --scope dstanesc

Wasm test

Interactive mode, eg. to check debugging info in the console

wasm-pack test --firefox

Headless mode

wasm-pack test --headless --firefox

Pack for local usage

npm pack pkg/node/
npm pack pkg/webpack/

Rust API

Function wasm_chunking_fastcdc::compute_chunks

pub fn compute_chunks(
    source: &[u8], 
    min_size: u32, 
    avg_size: u32, 
    max_size: u32
) -> Result<Vec<u32>, RangeError>

Compute chunks from a given slice of bytes.

The min_size specifies the preferred minimum chunk size, max_size the preferred maximum chunk size; the avg_size is the desired “normal size” of the chunks. The smallest acceptable min_size is 64 bytes and likewise 256 bytes and 1024 bytes for avg_size and respectively max_size

A js_sys::RangeError is returned when the above chunking specification is out of range.

Example:

use wasm_chunking_fastcdc::compute_chunks;
let data: Vec<u8> = br"Lorem ipsum dolor sit amet, consectetur adipiscing elit...put more bits in here...".to_vec();
let slice: &[u8] = &data;
let min_size: u32 = 64;
let avg_size: u32 = 256;
let max_size: u32 = 1024;
let offsets: Vec<u32> = compute_chunks(slice, min_size, avg_size, max_size).unwrap();

License

Licensed under either Apache 2.0 or MIT at your option.