unstorage

🌍 💾 Universal Storage Layer

Why ❓

Typically, we choose one or more data storages based on our use-cases like a filesystem, a database like Redis, Mongo, or LocalStorage for browsers but it will soon start to be lots of trouble for supporting and combining more than one or switching between them. For javascript library authors, this usually means they have to decide how many platforms they support and implement storage for each.

💡 Unstorage solution is a unified and powerful Key-Value (KV) interface that allows combining drivers that are either built-in or can be implemented via a super simple interface and adding conventional features like mounting, watching, and working with metadata.

Comparing to similar solutions like localforage, unstorage core is almost 6x smaller (28.9 kB vs 4.7 kB), using modern ESM/Typescript/Async syntax and many more features to be used universally.

📚 Table of Contents

• Usage
• Storage Interface
  • storage.hasItem(key)
  • storage.getItem(key)
  • storage.setItem(key, value)
  • storage.removeItem(key, removeMeta = true)
  • storage.getMeta(key, nativeOnly?)
  • storage.setMeta(key)
  • storage.removeMeta(key)
  • storage.getKeys(base?)
  • storage.clear(base?)
  • storage.dispose()
  • storage.mount(mountpoint, driver)
  • storage.unmount(mountpoint, dispose = true)
  • storage.watch(callback)
• Utils
  • snapshot(storage, base?)
  • restoreSnapshot(storage, data, base?)
  • prefixStorage(storage, data, base?)
• Storage Server
• Drivers
  • fs (node)
  • localStorage (browser)
  • memory (universal)
  • http (universal)
  • redis
• Making custom drivers
• Contribution
• License

Usage

Install unstorage npm package:

yarn add unstorage
# or
npm i unstorage

import { createStorage } from 'unstorage'

const storage = createStorage(/* opts */)

await storage.getItem('foo:bar') // or storage.getItem('/foo/bar')

Options:

• driver: Default driver (using memory if not provided)

Storage Interface

storage.hasItem(key)

Checks if storage contains a key. Resolves to either true or false.

await storage.hasItem('foo:bar')

storage.getItem(key)

Gets the value of a key in storage. Resolves to either string or null.

await storage.getItem('foo:bar')

storage.setItem(key, value)

Add/Update a value to the storage.

If the value is not a string, it will be stringified.

If value is undefined, it is same as calling removeItem(key).

await storage.setItem('foo:bar', 'baz')

storage.removeItem(key, removeMeta = true)

Remove a value (and it's meta) from storage.

await storage.removeItem('foo:bar')

storage.getMeta(key, nativeOnly?)

Get metadata object for a specific key.

This data is fetched from two sources:

• Driver native meta (like file creation time)
• Custom meta set by storage.setMeta (overrides driver native meta)

await storage.getMeta('foo:bar') // For fs driver returns an object like { mtime, atime, size }

storage.setMeta(key)

Set custom meta for a specific key by adding a $ suffix.

await storage.setMeta('foo:bar', { flag: 1 })
// Same as storage.setItem('foo:bar$', { flag: 1 })

storage.removeMeta(key)

Remove meta for a specific key by adding a $ suffix.

await storage.removeMeta('foo:bar',)
// Same as storage.removeMeta('foo:bar$')

storage.getKeys(base?)

Get all keys. Returns an array of strings.

Meta keys (ending with $) will be filtered.

If a base is provided, only keys starting with the base will be returned also only mounts starting with base will be queried. Keys still have a full path.

await storage.getKeys()

storage.clear(base?)

Removes all stored key/values. If a base is provided, only mounts matching base will be cleared.

await storage.clear()

storage.dispose()

Disposes all mounted storages to ensure there are no open-handles left. Call it before exiting process.

Note: Dispose also clears in-memory data.

await storage.dispose()

storage.mount(mountpoint, driver)

By default, everything is stored in memory. We can mount additional storage space in a Unix-like fashion.

When operating with a key that starts with mountpoint, instead of default storage, mounted driver will be called.

import { createStorage } from 'unstorage'
import fsDriver from 'unstorage/drivers/fs'

// Create a storage container with default memory storage
const storage = createStorage({})

storage.mount('/output', fsDriver({ base: './output' }))

//  Writes to ./output/test file
await storage.setItem('/output/test', 'works')

// Adds value to in-memory storage
await storage.setItem('/foo', 'bar')

storage.unmount(mountpoint, dispose = true)

Unregisters a mountpoint. Has no effect if mountpoint is not found or is root.

await storage.unmount('/output')

storage.watch(callback)

Starts watching on all mountpoints. If driver does not supports watching, only emits even when storage.* methods are called.

await storage.watch((event, key) => { })

Utils

snapshot(storage, base?)

Snapshot from all keys in specified base into a plain javascript object (string: string). Base is removed from keys.

import { snapshot } from 'unstorage'

const data = await snapshot(storage, '/etc')

restoreSnapshot(storage, data, base?)

Restore snapshot created by snapshot().

await restoreSnapshot(storage, { 'foo:bar': 'baz' }, '/etc2')

prefixStorage(storage, data, base?)

Create a namespaced instance of main storage.

All operations are virtually prefixed. Useful to create shorcuts and limit access.

import { createStorage, prefixStorage } from 'unstorage'

const storage = createStorage()
const assetsStorage = prefixStorage(storage, 'assets')

// Same as storage.setItem('assets:x', 'hello!')
await assetsStorage.setItem('x', 'hello!')

Storage Server

We can easily expose unstorage instance to an http server to allow remote connections. Request url is mapped to key and method/body mapped to function. See below for supported http methods.

🛡️ Security Note: Server is unprotected by default. You need to add your own authentication/security middleware like basic authentication. Also consider that even with authentication, unstorage should not be exposed to untrusted users since it has no protection for abuse (DDOS, Filesystem escalation, etc)

Programmatic usage:

import { listen } from 'listhen'
import { createStorage } from 'unstorage'
import { createStorageServer } from 'unstorage/server'

const storage = createStorage()
const storageServer = createStorageServer(storage)

// Alternatively we can use `storage.handle` as a middleware
await listen(storage.handle)

Using CLI:

npx unstorage .

Supported HTTP Methods:

• GET: Maps to storage.getItem. Returns list of keys on path if value not found.
• HEAD: Maps to storage.hasItem. Returns 404 if not found.
• PUT: Maps to storage.setItem. Value is read from body and returns OK if operation succeeded.
• DELETE: Maps to storage.removeIterm. Returns OK if operation succeeded.

Drivers

fs (node)

Maps data to the real filesystem using directory structure for nested keys. Supports watching using chokidar.

This driver implements meta for each key including mtime (last modified time), atime (last access time), and size (file size) using fs.stat.

import { createStorage } from 'unstorage'
import fsDriver from 'unstorage/drivers/fs'

const storage = createStorage({
  driver: fsDriver({ base: './tmp' })
})

Options:

• base: Base directory to isolate operations on this directory
• ignore: Ignore patterns for watch
• watchOptions: Additional chokidar options.

localStorage (browser)

Store data in localStorage.

import { createStorage } from 'unstorage'
import localStorageDriver from 'unstorage/drivers/localstorage'

const storage = createStorage({
  driver: localStorageDriver({ base: 'app:' })
})

Options:

• base: Add ${base}: to all keys to avoid collision
• localStorage: Optionally provide localStorage object
• window: Optionally provide window object

memory (universal)

Keeps data in memory using Set.

By default it is mounted to top level so it is unlikely you need to mount it again.

import { createStorage } from 'unstorage'
import memoryDriver from 'unstorage/drivers/memory'

const storage = createStorage({
  driver: memoryDriver()
})

http (universal)

Use a remote HTTP/HTTPS endpoint as data storage. Supports built-in http server methods.

This driver implements meta for each key including mtime (last modified time) and status from HTTP headers by making a HEAD request.

import { createStorage } from 'unstorage'
import httpDriver from 'unstorage/drivers/http'

const storage = createStorage({
  driver: httpDriver({ base: 'http://cdn.com' })
})

Options:

• base: Base URL for urls

Supported HTTP Methods:

• getItem: Maps to http GET. Returns deserialized value if response is ok
• hasItem: Maps to http HEAD. Returns true if response is ok (200)
• setItem: Maps to http PUT. Sends serialized value using body
• removeIterm: Maps to DELETE
• clear: Not supported

redis

Store data in a redis storage using ioredis.

import { createStorage } from 'unstorage'
import redisDriver from 'unstorage/drivers/redis'

const storage = createStorage({
  driver: redisDriver({
     base: 'storage:'
  })
})

Options:

• base: Prefix all keys with base
• url: (optional) connection string

See ioredis for all available options.

lazyConnect option is enabled by default so that connection happens on first redis operation.

Making custom drivers

It is possible to extend unstorage by creating custom drives.

• Keys are always normalized in foo:bar convention
• Mount base is removed
• Returning promise or direct value is optional
• You should cleanup any open watcher and handlers in dispose
• Value returned by getItem can be a serializable object or string
• Having watch method, disables default handler for mountpoint. You are responsible to emit event on getItem, setItem and removeItem.

See src/drivers to inspire how to implement them. Methods can

Example:

import { createStorage, defineDriver } from 'unstorage'

const myStorageDriver = defineDriver((_opts) => {
  return {
    async hasItem (key) {},
    async getItem (key) {},
    async setItem(key, value) {},
    async removeItem (key) {},
    async getKeys() {},
    async clear() {},
    async dispose() {},
    // async watch(callback) {}
  }
})

const storage = createStorage({
  driver: myStorageDriver()
})

Contribution

• Clone repository
• Install dependencies with yarn install
• Use yarn dev to start jest watcher verifying changes
• Use yarn test before pushing to ensure all tests and lint checks passing

License

MIT