KV Storage

A simple on disk key value storage for nodejs

Description

The implementation is done using an append only log where every value is stored in binary format and the keys are pointers to disk segments, similar to the bitcask engine, so the keys should fit in memory.

Installation

npm i @msiviero/kv-storage

Usage

• Basic usage
• Log compaction
• Iterate records

Basic usage

import { FileSystemStorage } from "@msiviero/kv-storage";

interface User {
  email: string;
  id: number;
}

// create a store. The directory will be created if doesn't exist
const storage = await FileSystemStorage.create<User>("directory to store data in");

await storage.put("mykey", {
  id: 1,
  email: "[email protected]",
});

/*
The result contains the stored data in the .data field, and a
metadata object containing:
- checksum: an md5 checksum of the stored buffer
- timestamp: timestamp of insertion
- key: the original key
*/
const result = await storage.get("mykey");

const user = result?.data;

Log compaction

The log compaction logic is left to the user, as it's different based on the application. To compact the log call the compact() method

await storage.compact();

Iterate records

// Using an async generator
for await (const it of storage.iterator()) {
 // do something with the record
 console.log(it);
}

// using a stream
storage
  .stream()
  .on("data", (result: Readonly<Result<User>>) => console.log(result))
  .on("end", () => console.log("finished"));