AnyWhichWay

A JavaScript schema optional extensible key-value backed datastore supporting element level security, triggers, query pipelines, graphs, objects, documents, joins and full-text indexing.

Introduction

AnyWhichWay can be backed by almost any key-value store that exposes get(key), set(key,data), del(key) or similar methods synchronously or asynchronously.

It is in an early BETA and currently supports:

1. Graph navigation over fully indexed data.

2. Object retrieval based on patterns.

3. Optional schema to support indexing and object validation.

4. Query joins.

5. Streaming analytics triggers for put/insert, patch/update, and delete.

6. "Smart" serialization. The database "learns" about new classes as they are inserted and restores data into appropriate class instances.

7. Over 30 piped array like query commands such as first(n), last(n), map(f), mapReduce(mapper,reducer) , pop(), reduce(f,init).

8. Custom graph navigation and piped commands in as little as one line of code.

9. Inspection and automatic "polyfilling" of passed in storage, e.g. storage can provide del, delete, remove, or removeItem.

10. Security using graph paths. This allows the application of security at any level desired, i.e. object, property, and even value.

11. Automatic data expiration using durations or specific dates.

12. Full text stemmed and tri-gram indexes and search.

This version has been tested with: localStorage and idbkvstore in the browser plus blockstore, kvvs, Redis, and scattered-store on the server. Addining testing with keyv in on the roadmap. AnyWhichWay has built in logic to find the appropriate get, set, delete, and clear methods.

The key value stores that will not work are those that generate file names for the keys without hashing, e.g. node-localstorage. This is because the keys generated internally by AnyWhichWay are often not valid file names.

Note: idbkvstore is currently very slow for put. In fact, things will run faster against a remote Redis store. This is a function of the underlying IndexedDB, which in our experience tends to be slow for simple key/value type tasks due but fast for more complex tasks.

Why AnyWhichWay

When processing data in JavaScript, one usually ends-up collecting things into arrays and then processing the arrays to sort, filter, reduce etc. This can consume large amounts of RAM and also means that result processing can't start until an array is fully populated, particularly with respect to simulating joins. AnyWhichWay provides a full set of array like functions that are actually backed by asynchronous generators. Even the join processor is an asynchronous generator that processes one tuple at a time. When processing large volumes of data, this ensures the non-blocking return of initial results as fast as possible while limiting RAM usage.

With AnyWhichWay, if you know the functions available on an array you know more than 80% of what is required to query and process data in a manner far more rich than many alternatives with a smaller footprint. It is currenlty about 50K uncompressed. Granted, AnyWhichWay is in BETA now, so there are also things missing, e.g. conflict resolution, full transaction management. We anticipate it will be about 25K compressed when PRODUCTION is released.

Contents

Introduction

Example Capabilities

Why AnyWhichWay

Assist Us

Installation

Documentation Notes

Starting A Database

Storing Data

Queyring Data

Graph Navigation

Inline Predicates

Query Patterns

Query Commands

Pipeline Step

Joins

Metadata

Schema

Security

Transactions

Triggers

Extending AnyWhichWay

Predicates

Release History

Example Capabilities

1. Graph navigation over fully indexed data supporting:

   a) exact matching, e.g. get("Person/email/'[email protected]'") retrieves the Person with the provided e-mail.

   b) inline tests, e.g. get("Person/age/gt(27)") retrieves all Persons over 27.

   c) existing value retrieval, e.g. get("Person/age") will return all the possible ages for Persons.

   d) wild cards, e.g. get("*/email/*"), will retrieve all objects that have an email property, not just Persons.

   e) inline functions, e.g. get("Person/age/(value) => value > 27 ? value) also retrieves all Persons over 27, although more slowly

   f) object methods, e.g. `get("Car/#/.makeAndModel()) will return all makeAndModel strings for all Cars.

2. Object retrieval based on patterns supporting:

   a) exact matching, e.g. get({email:"[email protected]"},Person) retrieves the Person with the provided e-mail.

   b) tests, e.g. get({age:value => value > 27 ? value,name:"Joe"},Person) retrieves all Persons over 27 with name "Joe".

   c) coerced classes, e.g. get({email:"[email protected]", instanceof:"Person"}) retrieves the Person with the provided e-mail.

3. Object joins supporting:

   a) patterns, e.g. join({instanceof:Guest},{instanceof:Guest},([guest1,guest2]) => guest1["#"]!=guest2["#"] ? [guest1,guest2] : false) retrieves all possible Guest pairs

   b) paths, e.g. join("Object/age/*","Object/age/*",([a,b]) => a.age && b.age && a.age!==b.age ? [a,b] : false) retrieves all pairs of differently aged Objects

4. Triggers:

   a) e.g. on('Object/optin/true',"set",({type,target,key,value}) => console.log(type,target,key,value) will log "put",<some object>,"optin",true whenever an object with the optin property set to true is added to the database.

5. Security using graph path strings or arrays:

   a) /@Object/<security rule> - controls all Objects

   b) /@Object/SSN/<security rule> - controls all SNN property data on all Objects

6. Data expiration:

   a) putObject({user:"[email protected]",passwordResetKey: 12345},30*60*1000}) - inserts a key for password reset that expires and is deleted in 30 minutes.

Assist Us

If you like the concept of AnyWhichWay give us a star here on GitHub or npmjs.

Help spread the word by a comment or up vote on echojs or a Tweet.

Installation

npm install anywhichway

AnyWhichWay will run in current versions of Chrome and Firefox.

Node v9.7.1 (the most recent at this writing) must be run with the --harmony flag.

Babel transpiled code will not work. Babel does not seem to generate correct asynchronous generators.

Documentation Notes

When "Object" is capitalized it refers to a direct instance of the class Object. When "object" is lower case it refers to an instance of any type of class except Array, which uses the term "array".

The property "#" is the default used for unique uuidv4 ids on objects in AnyWhichWay. See the section Metadata for more info.

The property "^" is the default used for object metadata. See the section Metadata for more info.

Starting A Database

Databases are instantiated with 2 arguments, a storage instance and an options object.

const mydb = new Database(localStorage,{inline:true,
                                        expirationInterval=10*1000})

or

const redis = redis.createClient({
		host:process.env.HOST,
		port:process.env.PORT,
		password:process.env.PWD,
		no_ready_check:true}),
	mydb = new Database(redis,{inline:true,promisify:true});

The storage argument can be anything that supports the localStorage API or something similar with get and set. If the storage argument is null an in-memory store is created.

The options object supports the following:

context - The object to use as the this scope for triggers and security rules. Defaults to null.

awaitIndexWrite:boolean - A flag to indicate if object insertion should return prior to writing the index. Defaults to true. Setting to false will speed-up inserts. Note: Objects are still written before the index, this just affects return behavior. Recommend setting to false when there is a slow network connection to the backing key-value store.

cacheStep:number - Sets the number of records to free if the cache reaches maxCache. If a float, then treated as a percentage. If an integer, treated as an absolute count. Defaults to 0.1 (10 percent).

expirationInterval:number - The number of milliseconds between each scan and deletion of expiring data. The default is 30 minutes, i.e. 30*60*1000. If <= 0, auto expiration is turned off. This can be changed at any time by setting <db>._options.expirationInterval.

getCurrentUser:function - The function to call to get the current userid. Although AnyWhichWay supports mappings between Users and Roles, it does not provide password management or an authentication mechanism. It is up to the developer to provide these capabilities, perhaps through OAuth, and support the delivery of an authenticated user id to the security mechanisms of AnyWhichWay. The function can take one argument, which is the database instance. It should return a Promise for an object with the property id, usually an e-mail address.

inline:boolean - A flag indicating whether or not to support and compile inline functions in graph paths. Defaults to 'false'. Warning: Setting to true opens up a code injection risk.

indexDates:boolean - Index dates by their component UTC milliseconds, seconds, minutes, etc. Off by default. Otherwise, just the seconds since January 1st, 1970 are indexed.

maxString:number - All strings are full text or trigram indexed, but those longer than maxString can only be matched using search. Defaults to 64.

maxCache:number - Sets max size of LFRU Cache. The default is 10,000.

promisify:boolean - If true, the passed in storage uses callbacks that will need to be wrapped with Promises.

textIndex:"stem"|"trigram"|* - Sets the full text index type. Use * to index both ways'

Storing Data

To use AnyWhichWay as a regular key/value store, just use an asynchronous version of the localStorage API, e.g.

await mydb.setItem("joe",{age:27,registered:true});
const joe = mydb.getItem("joe");
await mydb.removeItem("joe");

To store an object and have it indexed, use the extended API putObject(object). Then use graph paths (see Queyring Data) or getObject(id), matchObject(pattern), and removeObject(patternOrId), e.g.

// uses plain old Objects
let joe = await mydb.putObject({name:"Joe",age:27,address:{city:"Seattle}});
joe = await mydb.getObject(joe["#"]);
for await(const person of mydb.matchObject({name:"Joe"})) { <do something> };
await removeObject({name:"Joe",age:27});

// inserts a Person
const p = new Person({name:"Joe",age:27});
await mydb.putObject({name:"Joe",age:27});
 
// inserts a Person by coercing the data
await mydb.putObject({name:"Joe",age:27,instanceof:"Person"});

For putObject, you can also set a duration in milliseconds or expiration date/time with a second argument:

// expires in approximately one year
await mydb.putObject({name:"Joe",age:27},365*24*60*60*1000); 

// expires Wed Aug 01 2040 00:00:00
await mydb.putObject({name:"Joe",age:27},new Date("2040/08/01"));

Note: Note the default expiration processing interval in AnyWhichWay is 30 minutes, so durations less than 30*60*1000 are not useful unless this is changed using the start-up option expirationInterval.

The indexes can be accessed like file paths and take the form:

<@classname>/<property>/<value>[/<property>/<value>...]

value - Value of the property. Strings are surrounded by "", e.g. /@Object/name/"Joe". This allows the indexed to be typed, i.e. @Object/age/27 vs /@Object/cutomerid/"123456".

[/<property>/<value>] - Optional property/value pairs are the nested objects, if any, e.g `/@Object/address/city/"Seattle".

See Graph Navigation below for how to use these paths.

Querying Data

Graph Navigation

Data can be retrieved using a graph path, e.g.:

mydb.query().get("@Object/address/city/*").all();

Paths can also be handed in as arrays, e.g.:

mydb.query().get(["@Object","address","city","*").all();

Unless you are using AnyWhichWay as a simple key value store, graph references generally start with a @ followed by a property and a value or another property if the value is itself an object, e.g.

{address:{city:"Bainbridge Island",state:"WA",
          zipcode:{base:98110,plus4:0000}}}

is matched by:

@Object/address/city/"Bainbridge Island"

@Object/address/zipcode/base/98110

Any location in the path can also be the * wildcard, a compiled inline test, a dynamic inline test, a ".." relative location, e.g.

/@Object/address/city/ - matches all Objects with an address and a city property and returns the Objects

/@Object/address/city - returns all city names for Objects that have an address property

/@Object/address/state/"WA"/ - return all objects with addresses in WA. Note the quotes around "WA". Strings must be quoted so that the index can distinguish between numbers and strings, e.g. 27 vs "27".

/@Object/address/state/in(["WA","OR"])/ - return all objects with addresses in WA or OR

/@Object/address/zipcode/base/between(98100,98199,true)/ - return all Objects with an address in the zipcode base range of 98100 to 98199

Note the use of the between predicate above, this is one of many predicates available:

Inline Predicates

gt(testValue) - True for values that are > testValue.

gte(testValue) - True for values that are >= testValue.

eq(testValue,depth) - True for values that are == testValue. If depth > 0, tests objects to the depth. Object having equal ids satisfy eq.

between(bound1,bound2,edges) - True for values that are between bound1 and bound2. Optionally allows the value to equal boundary.

outside(bound1,bound2) - True for values that are outside bound1 and bound2.

eeq(testValue) - True for values that are === testValue`.

echoes(testValue) - True for values that sound like testValue.

matches(testValue) - True for values that match testValue where testValue is a RegExp. If testValue is a string it is converted into a RegExp.

contains(testValue) - True for values where value.indexOf(testValue)>=0 so it works on strings and arrays.

neq(testValue) - True for values that are != testValue.

lte(testValue) - True for values that are <= testValue.

lt(testValue) - True for values that are < testValue.

in(testValue) - True for values where testValue.indexOf(value)>=0 so it works on strings and arrays.

nin(testValue) - True for values where testValue.indexOf(value)===-1 so it works on strings and arrays.

not(f) - True for values where !f(value) is truthy.

type(testValue) - True for values where typeof(value)===testValue.

Tertiary nodes after the "#" selector can be property names or method calls, e.g.

@Car/#/model - gets all model names for all Cars doing a table scan, Car/model is faster.

@Car/#/.makeAndModel() - However, method names can only be invoked as a result of a table scan.

It should not be overlooked that by design graph paths can be escaped and passed directly over a network as get requests!

Arrays

Values stored in arrays can be accessed directly as children or via an array offset, e.g.

{
	name: "Joe",
	favoriteColors: ["red","green"]
}

/@Object/favoriteColors/"red"/ - Returns Objects where "red" is anywhere in the array.

/@Object/favoriteColors/0/"red"/ - Returns Objects where "red" is the first element.

Dynamic Inline Tests

/@Object/address/zipcode/base/(value) => value>=98100 && value<=98199) - return all Objects with an address in the zipcode base range of 98100 to 98199

/@Object/*/(value) => ... some code/ - a "table scan" across all Objects and all properties, returns all Objects with property values satisfying the inline

/@Object/#/(value) => ... some code/ -a "table scan" across all Objects, returns all Objects satisfying the inline

*/*/(value) => ... some code/ - a "table scan" across an entire database, returns all objects satisfying the inline

*/#/(value) => ... some code/ - a "table scan" across instances of all classes, returns all objects satisfying the inline

Notes: Dynamic in-line tests MUST use parentheses around arguments, even if there is just one.

Dynamic in-line tests expose your code to injection risk and must be enabled by setting inline to true in the options object when a database connection is created. Any in-line test can be added as a compiled test to avoid this issue. See Extending AnyWhichWay.

For convenience, the following are already defined:

Query Patterns

Query patterns are objects. If the query pattern is an instance of a specific kind of object, then only those kinds of objects will be matched.

Property values in query patterns may be literals or functions that return 'falsy' if they should fail.

// yield all Person's over 27 in the state of Washington.
mydb.query().get({age:value => value > 27,
                  address:{state:WA},
                  instanceof:Person}).all(); 

Query Commands

Queries are effectively a series of pipes that change the nature of the results or produce other side-effects. Internally, all query commands are wrapped in a generator functions that yield each result to the next command.

Queries take the forms:

<db>.query().<edgeYield>[.<edgeYield>,...<valueYield>[.<valueYield>|<pipelineStep>...[.<valueYield>|<pipelineStep>].<invoker>]`;

<db>.query().<valueYield>[.<valueYield>|<pipelineStep>...[.<valueYield>|<pipelineStep>].<invoker>]`; 
// only `provide` can be used as a `valueYield` when no edge is yielded first.

For example:

mydb.query().get("Object/age/*").value().filter(object => object.age>27).all();

.get("Object/age/*") - yields all edges for Objects with ages.

.value() - extracts the value from each edge.

.filter(object => object.age>=27) - filters for all objects with age >= 27.

.all() - executes the query and returns a promise for an array of objects.

Edge Yielding Commands

get(pathOrPattern) - get always yields an edge, except when the last portion is /#/.<methodCallOrPropertyName>, in which case it yields a value. See Query Patterns below.

Value Yielding Commands

delete(pathOrPatternOrId) - Deletes the specified item. Or, if no argument is provided, deletes the item yielded by the previous step in the query. Yields no result.

edge(raw) - Yields the edge itself as a JSON value. If raw is true, yields the generator that represents the edge.

edges(raw) - Yields the child edges of the edge itself as JSON values. If raw is true, yields the generators that represents the children.

fetch(url,options) - If an argument is provided, fetches the url with the provided options using the value it is passed as the body. If no arguments are provided, assumes the passed value will be a string representing a URL or an object of the form {url,options}. Also available as a Pipeline Step.

fork() - Creates a separate worker or Node process to run the command. NOT YET IMPLEMENTED.

join(...pathsOrPatterns,test) - Yields arrays of values combinations that satisfy test(<array combination>). See Joins below.

keys() - Yields the keys on all values that are objects. Any non-object values result in no yield. Also available as a Pipeline Step.

provide(...values) - Yields no values except those in values. Usually used directly after <db>.query(), often for testing purposes.

random(float) - Yields a random percentage of the values it is passed. The random method may also be used as part of a process step sequence. Also available as a Pipeline Step.

render(template) - Renders each value provided into the template and yields {value,string}. The template should use JavaScript template literal type substitution. The scope is bound to the value passed in. The render method may also be used as part of a process step sequence. Also available as a Pipeline Step.

search(keywordString,options) - The keywordString is tokenized and stems are used to search an index. After candidate documents are returned, exact matching of tokens is applied. The options object can take the flags {percentage:float,exact:true|false,stem:true|false,path:pathString,index:"stem"|"trigram"} If percentage is set, the provided percentage of tokens must be matched; otherwise, just one token is sufficient. If stem is true, then the candidate documents just have stems checked. If exact is true, then percentage and stem are ignored and there must be an exact match for the keywordString. By default all properties of an object are searched. If path is set as a dot or slash delimited string, then just the property or sub-property matching the path is searched. index specifies which index to use. It defaults to the database options property textIndex. If textIndex is * (both), search defaults to "stem".

value - Yields the value on the edge.

values(objectsOnly) - Yields all the values of properties on any objects passed down or primitive values. If objectsOnly is true, then any primitive values from edges are not yielded. Also available as a Pipeline Step.

Pipeline Step

Pipeline steps are almost identical to the calling interface of an array, with some enhancements.

aggregate([pattern,]aggregates) - Behaves similar to MongoDB aggregate except uses AnyWhichWay functional pattern matching and aggregators. See matches for functional pattern matching. The pattern {} matches all objects. You can also exclude the pattern argument and its is assumed to be {}, which will match all objects passed down the pipeline. The aggregates specification is an object the properties of which are the keys to group by. Their values are also objects naming the property to aggregate which are also objects naming the aggregated field and defining the aggregator functions. Aggregated filed names must be unique across the entire aggregate. The functional aggregators are simply reduce functions that take the accumulated value, the next value, and the aggregation as arguments. Aggregators are called in the order they are specified, so results can be based on aggregates computed earlier. The count property is always available on the aggregation. The first time aggregators are called the accumulator will be undefined. You can default it to 0 the first value with a single short statement, e.g. see the definition of sum and average below which will sum and average the balance across all objects.

aggregate({},{
	{
		"*" : {
			balance: {
				total: (accum,next) => accum===undefined ? next : accum + next,
				average: (accum,next,aggregation) => aggregation.total/aggregation.count,
			}
		}
	}
});

The slightly modified example will group by customer id:

aggregate({},{
	{
		"custid" : {
			balance: {
				total: (accum,next) => accum===undefined ? next : accum + next,
				average: (accum,next,aggregation) => aggregation.total/aggregation.count,
			}
		}
	}
});

concat(...args) - Adds the args to the end of the yielded query results.

collect() - Collects values into an array.

distinct(key,objectsOnly) - Yields only unique values. The key argument is optional and will yield only unique values for key when objects are passed in. If objectsOnly is falsy, primitive values will be also be yielded.

every(f) - Yields all values so long as every value satisfies f(value).

filter(f) - Only yields values for which f(value) returns truthy.

flatten() - Assumes an array is being passed from upstream and yields each value separately. If it is not an array, just yields the value.

forEach(f) - Calls f(value,index) for all values.

group(grouping,flatten) - grouping is an array of fields to group by. flatten will force the return of an array; otherwise an object of the form ... is returned.

map(f) - Calls f(value,index) and yields the value returned for each value it receives. f should return undefined to prevent anything from being passed down the processing chain.

mapReduce(map,reduce[,{key:string,value:string}]) - Takes upstream value as query result and then behaves like MongoDB mapReduce to yields the results as {key:string,value:any} pairs. The optional third argument supports the replacement of the key and value property names with custom names.

matches(pattern,value) - If pattern a primitive yields value if pattern===value. If pattern is a function value is yielded if pattern(value) is truthy. If pattern is an object, recursively calls matches on each property value of pattern and value.

pop(f) - Pulls the last value and does not yield it. If f is not null calls f(value) with the popped value.

push(value) - Yields value into the end of the results.

reduce(f,initial) - Yields the result of <all values>.reduce(f,initial).

reduceRight(f,initial) - Yields the result of <all values>.reduceRight(f,initial).

reverse() - Yields results in reverse order.

seen(f) - Calls f(value) once for each unique value received but yields all values.

slice(begin,end) - Yields just the values between the index begin and end. Using negative values uses offsets from end of yield process.

shift(f) - Does not yield the first value. If f is not null, calls f(value) with the shifted value.

some(f) - Yields all values so long as some value satisfies f(value).

sort(f) - Yields values in sorted order based on f. If f is undefined, yields in alpha sorted order.

split(...queryFragmentFunctions) - creates multiple yield streams with copies of the value passed to split. If an array is passed and it contains objects, each object is also copied. A queryFragmentFunction takes the form:

// "this" takes the place of "<database>.query()"
// there should be no "all()" or "exec()" at the end.
function() { return this.<command 1>.<command 2>...<command n>; }

test(test,value) - If test is a primitive or a RegExp it is treated as a RegExp; otherwise, it should be a function. Yields value if test(value) or test.test(value) is truthy.

splice(start,deleteCount,...items) - Yields items values after values up to start have been yielded and up to deleteCount items have not been yielded.

unshift(value) - Add value to the end of the yield stream.

Invokers

all() - Returns a promise for the values of a query as an array.

exec() - Processes a query without collecting any results. Conserves memory as a result.

Full Text Stemmed and Trigram Indexing and Search

To be written. For now see search in API documentation above.

Joins

join(...pathsOrPatterns,test) - Yields arrays of value combinations that satisfy test(<array combination>).

By convention you should destructure the argument to test. The test below will only yield combinations where the names are identical:

[object1,object2] => 
  object1.name && object2.name && object1.name===object2.name;

Metadata

The signature of metadata is: {created:Date,updated:Date,expires:Date,v:number}.

v is the version number.

Unique object uuidv4 ids are stored on objects themselves under the key # rather than in metadata. Their signature is: <classname>@<uuidv4>.

Dates have the id signature Date@<milliseconds>.

Schema

Schemas are optional with AnyWhichWay. They can be used to:

1. Validate data

2. Ensure referential integrity across objects/documents

3. Support records retention and expiration

4. Provide declarative security

By convention schemas are defined as static mmembers of classes/constructors. However, they can also be declared when registering classes using <database>.register(ctor,name,schema).

The surface of a schema may contain any of the following properties:

• index

• integrity

• management

• security

• validation

Each should point to an object as described below:

Index

If no schema index definition is provided, all properties on an object are indexed. If a property name is not listed it is indexed by default.

class Person() {

}
Person.schema = {
	index: {
		SSN: false
	}
}

Integrity - NOT YET IMPLEMENTED

Supports cascading deletes and the creation of reflexive relationships. Validates the arity (n-ary nature) of relationships.

The integrity value should be an object with a property for every case where an instance is expects to have another object as a value.

class Person() {

}
Person.schema = {
	integrity: {
		address: {
			kind: "heirarchical"
			arity: 1
		},
		favoriteColors: {
			kind: "heirarchical"
			arity: 3
		},
		siblings: {
			kind: "relational",
			arity: Infinity,
			reflect: "siblings",
			cascade: ["update"]
		}
	}
}

A person might then look like this:

{
	address: { ... },
	favoriteColors: [ ... ],
	siblings: [ ...ids ]
}

Management

Sets default durations. Unspecified durations are effectively Infinty. Turns off time stamping and versioning. By default creation and update time stamps are created and versions are incremented.

class Person() {
	
}
Person.schema = {
	management: {
		duration: milliseconds,
		timestamp: true|false,
		version: true|false
	}
}

Security - NOT YET IMPLEMENTED

Person.schema = {
	security: {
		SSN: [
			{group:"admin",read:true,write:true},
			{group:"owner",read:true,write:true} // owner is a special group
		],
		[key => true]: [
			
		]
	}
}

Validation

The validation value should be a function without closure scope that validates an entire object and returns a possible empty array of errors, or an object that has the same structure as the expected class instance, including nested objects. However, each property value should either be:

1. a function without closure scope that when passed a value, key, object from an instance returns true for valid or an Error object if invalid.

2. a validation specification taking the form:

{
	type: "boolean"|"number"|"string"|"object"|"<classname>",
	default: <value>|function,
	required: true|false,
	in: [<value>[,<value>...]],
	nin: [<value>[,<value>...]],
	matches: string, // converted to RegExp,
	between: [number,number],
	outside: [number,number],
	... any other defined test predicate
}

Nested objects can either be validated by a single function, or can have functions and validation specifications on nested object properties.

For example:

class Person() {
	constructor(config) {
		Object.assign(this,config);
	}
}
Person.schema = {
	validation: {
		name: value => typeof(value)==="string" || new TypeError("Person.name is required and must be a string"),
		age: value => value===undefined || typeof(value)==="number" || new TypeError("Person.age must be a number"),
		optin: {type:"boolean",default:false}
	}
}

Security

In addition to schema based security, security can expressed using graph paths and a special command secure(path,action,function). This allows the application of security at any level desired, e.g.

<db>.secure("Object","set",<security rule>) - controls set for all Objects and their child paths.

<db>.secure("Object/SSN","get",<security rule>) - controls get of all data stored in SNN property on all Objects

Security rules are just a special type of function receiving a security event object:

({key,path,action:"delete"|"get"|"set",value[,original]}) => ... your code ... 

Note: Wild cards and inline functions are NOT supported in security paths; however, heirarchies are supported. In the example above, both of the rules will be applied to Objects that contain an SSN property since the first rule applies to ALL Objects.

The key will be the full path to the data being accessed. path will be the current level in the security heirarchy.

You are free to modify value to obscure data or restore it from the original. As a result, security rules can be very powerful, you could examine a value to see if it looked like an SSN regardless of the key it is stored under and return a masked version.

The original field will be present when using set and contains the current values on disk. This allows unmasking of previously masked data prior to save. Changes to original are ignored.

At the moment it is up to the implementor to look-up session ids, user ids and groups using the if they are needed. The rules this will be bound to will be the value of the context property of the options used to start the database.

Returning undefined, will prevent the action.

Transactions

Writes of objects that do not contain nested objects or that contain only nested objects of classes that are not registered with schema are atomic and done before indexing. This is somewhat transactional; however, if there is a failure, indexes may be incomplete.

Writes of objects that contain nested objects that are registered as schema are not currently atomic. Each object is a separate atomic write. If the system fails between these writes, then there could be an inconsistent state and definitely an incomplete index.

All objects are written prior to indexing. Currently schema must be registered every time a database is started using the register(ctor,name=ctor.name,schema=ctor.schema) method on the database. Passing true as the value for schema will force indexing, but no validation will be done on the instances of ctor. As can be seen from the signature for register, by convention schema are defined as static properties on class definitions; however, this can be changed simply by calling register with three arguments.

The above being said, a base design for full transactional capability has been envisioned and transactional integrity is possible.

Triggers

<db>.on(key,eventType,callback) - Ads callback as a trigger on key for the event types set, delete.

The callback is invoked with an event object of the below form.

{key,action:"set"|"delete",value:<any value>,oldvalue:<any value>[,object,key]}`

The optional members object and key will only be present if the value at the path is an object.

Thecallback will be serialized between database sessions. So, it should not contain any closure scoped variables. Its this will be bound to the value of the context property of the options used to start the database.

Extending AnyWhichWay

Predicates

Predicates, e.g. gt, gte, can be added using the tests property on the options object when starting a database.

The below example woud just replace the outside predicate with what happens to be the same code as its internal representation.

function outside(b1,b2) {
	return value => value != b1 && value !=b2 && (b2 > b1 ? value<b1 || value>b2 : value>b1 || value<b2);
}
const db = new Database(store,{tests:{outside}});

Note: The function outside returns a function that take a single argument, value. This function is invoked by AnyWhichWay with the current value of the data item being tested.

Benchmarks

Proper benchmarking is udner development; meanwhile, below are the typical times to run the unit tests:

Redis (remote high speed connection): 10s Redis (remote low speed connection): 12s localStorage: 5s idbKVStore: 16s scattered-store: 2s Blockstore: 450ms KVVS Speed Optimize: 1s KVVS Memory Optimized: 2s

Release History (reverse chronological order)

2018-04-23 - BETA v0.0.27b Eliminating test data pushed with earlier build.

2018-04-20 - BETA v0.0.26b Tested with KVVS (Key Value Versioned Store).

2018-04-16 - BETA v0.0.25b Added user and role creation. Fixed issues with instanceof.

2018-04-13 - BETA v0.0.24b Enhanced documentation. Added to start-up option awaitIndexWrite.

2018-04-13 - BETA v0.0.23b Implemented MongoDB like mapReduce. Allow null storage for memory only database. Renamed fork to split. Created multi-process fork. Enhanced schema support.

2018-04-08 - ALPHA v0.0.22a Major re-write of internals. 20% smaller. 2-3x faster. Index names prefixed with @. Added a type of LFRU Cache. split fixed. Next release will be a BETA.

2018-04-01 - ALPHA v0.0.21a Predicates in the query pipleine have been deprecated. Use map instead. last(n) has been deprecated, use slice(-n) instead. first(n) has been deprecated, use slice(0,n) instead. patch(pattern,data,expires) has been deprecated, use get(pattern).assign(object,true,expires) instead. NOTE: relative paths broken.

2018-03-22 - ALPHA v0.0.20a full text search added, NOTE: relative paths and split broken.

2018-03-17 - ALPHA v0.0.19a object patching now implemented

2018-03-16 - ALPHA v0.0.18a enhanced documentation.

2018-03-16 - ALPHA v0.0.17a moved triggers to their own partition for performance.

2018-03-16 - ALPHA v0.0.16a enhanced documentation, added ondelete handling.

2018-03-14 - ALPHA v0.0.15a enhanced documentation.

2018-03-14 - ALPHA v0.0.14a tested with blockstore, improved split.

2018-03-14 - ALPHA v0.0.13a fixed some string escape issues for idbkvstore.

2018-03-13 - ALPHA v0.0.12a enhanced documentation.

2018-03-13 - ALPHA v0.0.11a enhanced documentation, added assign, default, fetch, fork.

2018-03-13 - ALPHA v0.0.10a enhanced documentation, added auto expiration, published to NPM.

2018-03-12 - ALPHA v0.0.9a enhanced documentation, published to NPM.

2018-03-11 - ALPHA v0.0.8a delete now working for objects, paths pending.

2018-03-11 - ALPHA improved metadata support, started work on delete.

2018-03-11 - ALPHA enhanced documentation, added events and instanceof support.

2018-03-09 - ALPHA enhanced documentation, added schema validation.

2018-03-08 - ALPHA enhanced documentation, improved relative paths and nested objects.

2018-03-07 - ALPHA Enhanced documentation, pattern matching, joins, and unit tests.

2018-03-04 - ALPHA Publication of major re-write and release track ALPHA.

License

MIT License

Copyright (c) 2018 Simon Y. Blackwell, AnyWhichWay, LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.