npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

level-path-index

v0.0.5

Published

leveldb indexes for properties of items at or anywhere below a point in a materialized path

Downloads

7

Readme

level-path-index

Build status

index properties of items that live in a tree of materialized paths - using levelup

installation

$ npm install level-path-index

what it does

Indexes key=value properties of an object against a materialised path so you can ask things like:

  • match all items that match 'color=red' and are descendents of /home/rodney
  • match all items that are direct children of /home/rodney

example

var level = require('level');
var sub = require('level-sublevel');
var pathindex = require('level-path-index');

var db = sub(level(__dirname + '/pathdb', {
	 valueEncoding: 'json'
}))

var treedb = db.sublevel('folders')

var treeindex = pathindex(treedb, '_treeindex', function(key, value, emit){

	// index multiple values for one field
	(obj.colors || []).forEach(function(color){

		// emit the 'path', 'field' and 'value'
		emit(key, 'color', color)
	})

	emit(key, 'name', obj.name)
})

Then stick some data in your leveldb:

treeindex.batch([{
	type:'put',
	key:'/home/rodney/catpictures/goofycat.jpg',
	value:{
	  name:'goofy 1',
	  colors:['red', 'blue'],
	  description:'this cat crazy ass stoopid',
	  otherstuff:'...'
	 }
}, {
	type:'put',
	key:'/home/rodney/shoppinglist/catfood.txt',
	value:{
  	name:'cat yum yums',
  	colors:['red', 'yellow'],
  	description:'wot I needs to feeds the feline overlord',
  	otherstuff:'...'
	}
}], function(err) {

	// data and indexes are inserted!

})

Now we can search for things that match color=red and live somewhere under '/home/rodney':

var through = require('through');

treeindex.descendentStream('/home/rodney', {
	color:'red'
}).pipe(through(function(doc){
	console.dir(doc);
}))

/*
{
	name:'goofy 1',
	colors:['red', 'blue'],
	...
}
{
	name:'cat yum yums',
	colors:['red', 'yellow'],
	...
}

*/

You can also find direct children of an entry and use multiple clauses to your query:

treeindex.childStream('/home/rodney/catpictures', {
	color:'red',
	name:'goofy 1'
}).pipe(through(function(doc){
	console.dir(doc);
}))

/*
{
	name:'goofy 1',
	colors:['red', 'blue'],
	...
}
*/

index structure

In a single query step - there are 3 parts in the combined index:

  • fieldname
  • value
  • tree location

Assuming this question:

find all descendents of '/a' where the color is red

Then we have:

  1. fieldname (color)
  2. value (red)
  3. tree location (/a/b/c)

If we create the index in this strict order then the key would become:

color~red~/a/b/c

descendent query

The first part of the query is ok - color=red - this would mean leveldb range like this:

{
	start:'dv~color~red~',
	end:'dv~color~red~\xff'
}

Because we are doing a (d)escendent query with some (v)alues - the key is prepended with 'dv'

Now to include the path - we are looking below '/a' - we add the splitter '/' on the end and the level range becomes:

{
	start:'dv~color~red~/a/',
	end:'dv~color~red~/a/\xff'
}

This would match our item - which is living 2 layers below (in '/a/b/c')

child query

Children is slightly different - to make child request fast an extra index is created.

This avoids loading all descendents of a top level node when all you want are its children.

The child index works by seperating the parent path from the node path.

For our example (color~red~/a/b/c) the following index would also be created:

cv~color~red~/a/b/~_~c

The child value indexes are prepended with 'cv'.

The split between the parent and child path means we can ask for children of '/a/b' and only the direct children are loaded.

{
	start:'cv~color~red~/a/b/~_~',
	end:'cv~color~red~/a/b/~_~\xff'
}

This would match '/a/b/c' (cv~color~red~/a/b/~_~c) as a direct child

Empty queries

You can also load descendents and children with a blank query - a seperate index for empty queries is used - the key for our example is simply:

/a/b/c

So if we just wanted descendents of '/a/b' we can use this range:

{
	start:'dt~/a/b/',
	end:'dt~/a/b/\xff'
}

The empty query indexes (t)ree for descendents are prepended with 'dt' and for children 'ct'

api

pathindex(db, [indexdb], mapper(key, value, emit))

Pass the document database, optionally the name/sublevel for the indexes and a mapper function that will index each document as it is updated

the mapper is run with the key and value of the update and an emit function.

emit is a function with a (path, field, value) signature and be called multiple times to add an index to the document.

var tree = pathindex(mydb, function(key, value, emit){
	emit(value.path, 'type', value.type)
	emit(value.path, 'capital', value.capital)
})

index.save(key, value, callback)

Insert a value for a key and create the indexes based on your mapper function.

tree.save('/uk/south/west/bristol, {
	name:'Bristol',
	type:'city',
	size:'medium'
}, function(err){
	
})

index.batch(arr, callback)

Insert an array of documents - this must be a list of leveldb batch commands e.g.:

tree.batch([
  {key: '/uk/south/west/bristol', value: {name:'Bristol', size: 'medium'}}, type: 'put'},
  {key: '/uk/south/east/london', value: {name:'London', size: 'large'}}, type: 'put'},
  {key: '/uk/north/east/newcastle', value: {name:'Newcastle', size: 'medium'}}, type: 'put'},
  {key: '/uk/north/west/liverpool', value: {name:'Liverpool', size: 'medium'}}, type: 'put'}
], function (err, batch) {

})

index.descendentStream(path, searchTerms)

Return a read stream for entries that live at or below the given path:

tree.descendentStream('/uk/north', {
	size:'medium'
}).pipe(through(function(doc){
	console.dir(doc.key);
}))

// /uk/north/east/newcastle
// /uk/north/west/liverpool

You can use multiple search terms and the values can be lists which all must match:

tree.descendentStream('/uk', {
	size:'medium',
	tags:['red','yellow']
}).pipe(through(function(){
	
}))

index.childStream(path, searchTerms)

This is the same as descendent stream but for entries directly below the given path:

tree.childStream('/uk/south/west').pipe(through(function(val){
	console.dir(val.path);
}))

// /uk/south/west/bristol

index.descendentKeyStream(path, searchTerms)

This is the same as descendent stream but will return only the keys of the matched items

tree.descendentKeyStream('/uk/south').pipe(through(function(val){
	console.dir(val);
}))

// /uk/south/west/bristol
// /uk/south/east/london

index.childKeyStream(path, searchTerms)

This is the same as child stream but will return only the keys of the matched items

tree.childKeyStream('/uk/south/west').pipe(through(function(val){
	console.dir(val);
}))

// /uk/south/west/bristol

pull streams

Each of the 4 read stream methods also come with pull-stream equivalents

index.descendentPullStream(path, searchTerms)

index.descendentKeyPullStream(path, searchTerms)

index.childPullStream(path, searchTerms)

index.childKeyPullStream(path, searchTerms)

license

MIT