npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

hollowdb-vector

v0.1.4

Published

A vectorDB over HollowDB using HNSW

Downloads

17

Readme

Installation

Install the package:

yarn add hollowdb-vector
pnpm add hollowdb-vector
npm install hollowdb-vector

Usage

You can create the VectorDB as follows:

import HollowDBVector from "hollowdb-vector";
import { WarpFactory, defaultCacheOptions } from "warp-contracts";
import { SetSDK } from "hollowdb";
import { Redis } from "ioredis";
import { RedisCache } from "warp-contracts-redis";

// connect to Redis
const redis = new Redis();

// create Warp instance with Redis cache
const warp = WarpFactory.forMainnet().useKVStorageFactory(
  (contractTxId: string) =>
    new RedisCache({ ...defaultCacheOptions, dbLocation: `${contractTxId}` }, { client: redis }),
);

// create HollowDB SDK
const wallet = JSON.parse(readFileSync("./path/to/wallet.json", "utf-8"));
const contractTxId = "your-contract-tx-id";
const hollowdb = new SetSDK<string>(wallet, contractTxId, warp);

// create HollowDB Vector
const vectordb = new HollowDBVector(hollowdb);

Inserting a Vector

With this, you can insert a new point:

const point = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];

// any object
const metadata = {
  name: "My favorite vector!",
};

// insert a point
await vectordb.insert(point, metadata);

Metadata is optional, and you can leave it out during insert. If you would like to set it a later time, you can always do:

vectordb.db.set_metadata(index, metadata);

[!NOTE]

The complexity of inserting a point may increase with more points in the DB.

Fetching a Vector

You can get a vector by its index, which returns its point value and metadata:

const { point, metadata } = await vectordb.get_vector(index);

Querying a Vector

You can make a query and return top K relevant results:

// a query point
const query = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];

// number of top results to return
const K = 10;

// make a KNN search
const results = await vectordb.knn_search(query, K);

// each result contains the vector id, its distance to query, and metadata
const { id, distance, metadata } = results[0];

Deploying your own Contract

HollowDB Vector exports a static function that allows you two deploy a new contract that you own. Assuming that you have a wallet and a warp instance as described above, you can create a new contract with:

const { contractTxId } = await HollowDBVector.deploy(wallet, warp);
console.log("Deployed at:", contractTxId);

Setup

For local setup of this repo, first clone it.

git clone https://github.com/firstbatchxyz/hollowdb-vector

Then, install packages:

pnpm install

Peer dependencies should be installed automatically.

Protobuffers

We include the pre-compiled protobuf within the repo, but if you were to change the protobuf later, you can generate the compiled code as follows:

# HNSW protobufs
pnpm proto:code:hnsw # generate js code
pnpm proto:type:hnsw # generate types

# Request protobufs
pnpm proto:code:req # generate js code
pnpm proto:type:req # generate types

Testing

Tests are ran over a few cases for a fixed set of $N, K$ parameters that are prepared in Python, and are compared in Typescript. Run the tests via:

pnpm test

[!WARNING]

You need a live Redis server for the HollowDB test to work. Furthermore, the HollowDB tests may take some time.

Styling

Check the formatting with:

pnpm format

Lint everything with:

pnpm lint

You can also check types with:

pnpm check

Legacy

HollowDB Vector replaces DANNY, for the legacy code please refer to this branch.

License

HollowDB Vector is licensed under Apache 2.0.