npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@zilliz/feder

v1.0.7

Published

visualization packages for vector space

Downloads

874

Readme

Feder

What is feder

Feder is an javascript tool that built for understanding your embedding vectors, feder visualizes faiss, hnswlib and other anns index files, so that we can have a better understanding how anns work and what are high dimensional vector embeddings.

So far, we are focusing on the Faiss (only ivf_flat) index file and HNSWlib (hnsw) index file, we will cover more index types later.

Feder is written in javascript, and we also provide a python library federpy, which is based on federjs.

NOTE:

  • In IPython environment, it supports users to generate the corresponding visualization directly.
  • In other environments, it supports outputting visualizations as html files, which can be opened by the user through the browser with web service enabled.

Online demos

How feder works

Wiki

HNSW visualization screenshots

image

IVF_Flat visualization screenshots

image image image

Quick Start

Installation

Use npm or yarn.

yarn install @zilliz/feder

Material Preparation

Make sure that you have built an index and dumped the index file by Faiss or HNSWlib.

Init Feder

Specifying the dom container that you want to show the visualizations.

import { Feder } from '@zilliz/feder';

const feder = new Feder({
  filePath: 'faiss_file', // file path
  source: 'faiss', // faiss | hnswlib
  domSelector: '#container', // attach dom to render
  viewParams: {}, // optional
});

Visualize the index structure.

  • HNSW - Feder will show the top-3 levels of the hnsw-tree.
  • IVF_Flat - Feder will show all the clusters.
feder.overview();

Explore the search process.

Set search parameters (optional) and Specify the query vector.

feder
  .setSearchParams({
    k: 8, // hnsw, ivf_flat
    ef: 100, // hnsw (ef_search)
    nprobe: 8, // ivf_flat
  })
  .search(target_vector);

Examples

We prepare a simple case, which is the visualizations of the hnsw and ivf_flat with 17,000+ vectors that embedded from VOC 2012).

git clone [email protected]:zilliztech/feder.git
cd feder
yarn install
yarn dev

Then open http://localhost:12355/

It will show 4 visualizations:

  • hnsw overview
  • hnsw search view
  • ivf_flat overview
  • ivf_flat search view

Feder for Large Index

Feder consists of three components:

  • FederIndex - parse the index file. It requires a lot of memory.
  • FederLayout - layout calculations. It consumes a lot of computational resources.
  • FederView - render and interaction.

In case of excessive amount of data, we support separating the computation part and running it on a node server. We have two solutions for you:

  • oneServer
    • federServer (with FederIndex and FederLayout).
  • twoServer
    • indexServer (with FederIndex)
    • layoutServer (with FederLayout)

Referring to case/oneServer and case/twoServer.

Example with One Server

  1. launch the server
yarn test_one_server_backend
  1. launch the front web service
yarn test_one_server_front
  1. open http://localhost:8000

Example with Two Servers

  1. launch the FederIndex server
yarn test_two_server_feder_index
  1. launch the FederLayout server
yarn test_two_server_feder_layout
  1. launch the front web service
yarn test_two_server_front
  1. open http://localhost:8000

Pipeline - explore a new dataset with feder

Step 1. Dataset preparation

Put all images to test/data/images/. (example dataset VOC 2012)

You can also generate random vectors without embedding for index building and skip to step 3.

Step 2. Generate embedding vectors

Recommend to use towhee, one line of code to generating embedding vectors!

We have the encoded vectors ready for you.

Step 3. Build an index and dump it.

You can use faiss or hnswlib to build the index.

(*Detailed procedures please refer to their tutorials.)

Referring to test/data/gen_hnswlib_index_*.py or test/data/gen_faiss_index_*.py

Or we have the index file ready for you.

Step 4. Init Feder.

import { Feder } from '@zilliz/feder';
import * as d3 from 'd3';

const domSelector = '#container';
const filePath = [index_file_path];
const source = "hnswlib"; // "hnswlib" or "faiss"

const mediaCallback = (rowId) => mediaUrl;

const feder = new Feder({
  filePath,
  source,
  domSelector,
  viewParams: {
    mediaType: 'img',
    mediaCallback,
  },
});

If use the random_data, no need to specify the mediaType.

import { Feder } from '@zilliz/feder';
import * as d3 from 'd3';

const domSelector = '#container';
const filePath = [index_file_path];

const feder = new Feder({
  filePath,
  source: 'hnswlib',
  domSelector,
});

Step 5. Explore the index!

Visualize the overview

feder.overview();

or visualize the search process.

feder.search(target_vector[, targetMediaUrl]);

or randomly select an vector as the target to visualize the search process.

feder.searchRandTestVec();

More cases refer to the test/test.js

Blogs

Roadmap

We're still in the early stages, we will support more types of anns index, and more unstructured data viewer, stay tuned.

Acknowledgments