hollowdb-vector
v0.1.4
Published
A vectorDB over HollowDB using HNSW
Downloads
9
Maintainers
Readme
Installation
Install the package:
yarn add hollowdb-vector
pnpm add hollowdb-vector
npm install hollowdb-vector
Usage
You can create the VectorDB as follows:
import HollowDBVector from "hollowdb-vector";
import { WarpFactory, defaultCacheOptions } from "warp-contracts";
import { SetSDK } from "hollowdb";
import { Redis } from "ioredis";
import { RedisCache } from "warp-contracts-redis";
// connect to Redis
const redis = new Redis();
// create Warp instance with Redis cache
const warp = WarpFactory.forMainnet().useKVStorageFactory(
(contractTxId: string) =>
new RedisCache({ ...defaultCacheOptions, dbLocation: `${contractTxId}` }, { client: redis }),
);
// create HollowDB SDK
const wallet = JSON.parse(readFileSync("./path/to/wallet.json", "utf-8"));
const contractTxId = "your-contract-tx-id";
const hollowdb = new SetSDK<string>(wallet, contractTxId, warp);
// create HollowDB Vector
const vectordb = new HollowDBVector(hollowdb);
Inserting a Vector
With this, you can insert a new point:
const point = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];
// any object
const metadata = {
name: "My favorite vector!",
};
// insert a point
await vectordb.insert(point, metadata);
Metadata is optional, and you can leave it out during insert
. If you would like to set it a later time, you can always do:
vectordb.db.set_metadata(index, metadata);
[!NOTE]
The complexity of inserting a point may increase with more points in the DB.
Fetching a Vector
You can get a vector by its index, which returns its point value and metadata:
const { point, metadata } = await vectordb.get_vector(index);
Querying a Vector
You can make a query and return top K relevant results:
// a query point
const query = [-0.28571999073028564 /* and many more... */, 0.13964000344276428];
// number of top results to return
const K = 10;
// make a KNN search
const results = await vectordb.knn_search(query, K);
// each result contains the vector id, its distance to query, and metadata
const { id, distance, metadata } = results[0];
Deploying your own Contract
HollowDB Vector exports a static function that allows you two deploy a new contract that you own. Assuming that you have a wallet
and a warp
instance as described above, you can create a new contract with:
const { contractTxId } = await HollowDBVector.deploy(wallet, warp);
console.log("Deployed at:", contractTxId);
Setup
For local setup of this repo, first clone it.
git clone https://github.com/firstbatchxyz/hollowdb-vector
Then, install packages:
pnpm install
Peer dependencies should be installed automatically.
Protobuffers
We include the pre-compiled protobuf within the repo, but if you were to change the protobuf later, you can generate the compiled code as follows:
# HNSW protobufs
pnpm proto:code:hnsw # generate js code
pnpm proto:type:hnsw # generate types
# Request protobufs
pnpm proto:code:req # generate js code
pnpm proto:type:req # generate types
Testing
Tests are ran over a few cases for a fixed set of $N, K$ parameters that are prepared in Python, and are compared in Typescript. Run the tests via:
pnpm test
[!WARNING]
You need a live Redis server for the HollowDB test to work. Furthermore, the HollowDB tests may take some time.
Styling
Check the formatting with:
pnpm format
Lint everything with:
pnpm lint
You can also check types with:
pnpm check
Legacy
HollowDB Vector replaces DANNY, for the legacy code please refer to this branch.
License
HollowDB Vector is licensed under Apache 2.0.