ixixx-js

This library implements trix text indexing file generation. Trix indexes allow you to search a large amount of free text data using static files and byte-range requests, no server side code needed

It is basically a translation of https://github.com/ucscGenomeBrowser/kent/blob/master/src/index/ixIxx/ixIxx.c from C into JS, plus some added code to keep memory usage low by doing an external disk based sort. The original C library basically loads all keywords into memory, and sorts the keys of a large hash, but this library ran into some node.js limits on doing this with larger files and so it uses the external sort to keep memory usage lower

CLI

npm install -g ixixx
ixixxjs yourfile.text [out.ix] [out.ixx]

Internal API

ixIxxStream(stream: Readable, outIxFilename: string, outIxxFilename:string)
ixIxx(inFilename:string,outIxFilename:string,outIxxFilename:string)

The trix concept

Takes an input file like this, containing a mapping of a keyword to several keywords e.g.

MyGene0001  kinase signalling
MyGene0002  binding zinc

And then crates a new file thats kind of like the inverse of it in an ix file

binding MyGene0002
kinase MyGene0001
signalling MyGene0001
zinc MyGene0002

And indexes it so that it has a prefix, and a byte offset to words starting with that prefix in an ixx file e.g. (made up numbers but conceptually something like this)

bindin0000000000
signal0000000114

So that when you type e.g. sig it can lookup where approximately you want to start looking in the ixx file and then perform byte range requests against the ix file, and find that you are looking for MyGene0001

See also

https://github.com/GMOD/trix-js for the client side library to do the searches