live-index
v3.0.1
Published
Maintain an index into a log file as it is appended to and rotated
Downloads
5
Readme
live-index
Maintain an index into a log file as it is appended to and rotated
Installation
npm install --save live-index
API
var LiveIndex = require("live-index")
Constructor LiveIndex([options])
Instantiates a LiveIndex
instance. The options
object takes the following properties:
pathToWatch
: The path to a log file you want to watch and index.indexer
: A function you define to update the index. See theindexer
delegagte documentation below for what gets passed to this function.tailChunkSize
: Set the size in bytes that should be read at a time from the data files. You can use this to control the maximum size of the chunks dispatched to your indexer function.rotationPollInterval
: Poll for data file rotation every specified number of milliseconds, instead of relying onfs.watch()
rename events. Try this if indexing stops after a rotation.
If both pathToWatch
and indexer
are defined, then .watch()
is automatically called for you.
Method setIndexer(indexer)
Sets indexer
as the indexer delegate. See the indexer
delegate documentation below for what gets passed to this function.
Method setIndexStorageObject(obj)
Sets the object to be used for in memory storage of index identifiers and file positions. This overrides the default Map
instance.
The object specified by the obj
parameter must have .get(id, [callback])
and .set(id, val, [callback])
methods. They may optionally be asynchronous; in which case they must either call callback
in the standard error-first manner, or return a promise.
Method setPathToWatch(path)
Set the path
of the file to watch once .watch()
is called.
Method watch()
Begin watching the files that have been added to the index.
Method addStaticDataFile(path, [callback])
Appends a file given by path
to the list of data files and indexes it. Calls the callback once indexing is complete. If callback
is not specified a promise is returned instead.
Method readStreamBetweenIndexes(startIdentifier, endIdentifier, [callback])
Asynchronously returns a Readable
stream starting from the data file and position specified by startIdentifier
and ends at the data file and position specified by endIdentifier
. This stream will automatically span across multiple data files. If any or both of the identifiers are not in the index, then undefined
is returned instead of a stream object. If the index for endIdentifier
comes before startIdentifier
, an error is thrown.
The callback
parameter is standard error-first. If it is omitted, a promise that resolves to the stream returned instead.
Delegate indexer(chunk, addIndex, processedTo)
Your indexer
function must handle these parameters:
chunk
: TheBuffer
object containing the data your indexer will look at to create indexes.addIndex
: A callback to create an index entry or link.processedTo
: A callback you can optionally call to indicate how much of the chunk you were able to process.
Callback: addIndex(identifier, location)
Call the addIndex()
function from your indexer to add entries to the index. There are two ways to call this function. In the first form, location
is an integer specifying the byte offset in the chunk
that the index entry represents.
In the second form, location
is a string, specifying another identifier that this index should link to. When you look up identifier
later, it will automatically be resolved to the index entry specified by location
.
Callback: processedTo(positionInChunk)
Calling this in your indexer function to tell the LiveIndex how much of the chunk you were able to process. The next time your indexer function is called, the passed chunk
will contain what you were unable to process.
Contributing
Patches are welcome! This module is written in ES2015. Use npm run compile
to compile the source. All changes should include a test case or modification to an existing test case. Use npm test
to run the tests.