JSDB

Provides mysql and neo4j agnosticators, data stashing and ingesting methods.

Instal

npm install @totemstan/jsdb	

Start

npm run start ?					# List start options

Manage

npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation

Usage

import * from "@totemorg/jsdb";

Program Reference

JSDB

Provides mysql and neo4j agnosticators. This module documented in accordance with jsdoc.

Requires: module:enums, module:cluster, module:os, module:fs
Author: ACMESDS

Dataset Dependencies

openv.hawks Queried for moderaters when journalling a dataset.
openv.journal	Updated with changes when journalling enabled.
openv.locks Updated when record locks used (e.g. using forms).
openv.files Databrick files when saving stashes
openv._stats Databrick stats when saving stashes
openv.events For storing event data during saving stashes
openv.profile Client information to manage task queues
openv.queues Task Queues managed by regulator
openv.cache Place to cache data

Env Dependencies

URL_MYSQL=http://$KEY_MYSQL@localhost:3306
URL_NEO4J=http://$KEY_NEO4J@localhost:7474
URL_TXMAIL=http://[email protected]:587
URL_RXMAIL=
URL_LEXNEX=https:$KEY_LEXNEX//services-api.lexisnexis.com/v1/  

Example

### Acquire JSDB and optionally configures its emitters and database
	const {neoThread, cyper, config} = JSDB = require("jsdb").config({ 
	
		emit:  (crude,parms) => {  // method to broadcast changes to other socket.io clients
		}, 
		
		mysql : {	// 	database connection parms
			host: ...
			user: ...
			pass: ...
		}

	});

Example

### Classic mysql access
	sqlThread( sql => {
	
		// classic query

		sql.query( "...", [ ... ], (err,info) => {
		});
		
		// crud helpers:

		sql.Index(ds, query, keys => { ... })
		sql.Delete(ds,where, (err,info) => { ... })
		sql.Update(ds,where,body,(err,info) => { ... })
		sql.Insert (ds,body,(err,info) => { ... } )
		sql.Select(ds, index, where, opts, (err,recs) => { ... })

		// there are also various enumerators and other utility functions.

	});

Example

### Somewhat experimental method to access mysql datasets by context
	sqlThread( sql => {
	
		sql.context( {ds1:ATTRIBUTES, ds2:ATTRIBUTES, ... }, ctx => {

			const {ds1,ds2, ... } = ctx;

		});
		
	});

where dsN are datasets having context ATTRIBUTES = {key:value, ... } described below. 

Using dataset contexts, JSDB permits queries of the form:

	ds.rec = { FIELD:VALUE, ... }			// update matched record(s) 
	ds.rec = [ {...}, {...}, ... ]			// insert record(s)
	ds.rec = null 							// delete matched record(s)
	ds.rec = function CB(recs,me) {...}		// select matched record(s)

or like this:

	ds.res = callback() { ... }
	ds.data = [ ... ]
	ds.rec = CRUDE

or in record-locked mode using:

	ds.rec = "lock.select"
	ds.rec = "lock.delete"
	ds.rec = "lock.update"
	ds.rec = "lock.insert"

Dataset ATTRIBUTES = { key: value, ... } provide SQL agnostication:

	table: 	DB.TABLE || TABLE
	where: 	[ FIELD, FIELD, ... ] | { CLAUSE:null, nlp:PATTERN, bin:PATTERN, qex:PATTERN, has:PATTERN, like:PATTERN, FIELD:VALUE, FIELD:[MIN,MAX], ...} | CLAUSE
	res: 	function CB(ds) {...}
	having: [ FIELD, VALUE ] | [ FIELD, MIN, MAX ] | {FIELD:VALUE, CLAUSE:null, FIELD:[MIN,MAX], ...} | CLAUSE
	order: 	[ {FIELD:ORDER, ...}, {property:FIELD, direction:ORDER}, FIELD, ...] | "FIELD, ..."
	group: 	[ FIELD, ...] | "FIELD, ..."
	limit: 	[ START, COUNT ] | {start:START, count:COUNT} | "START,COUNT"
	index:	[ FIELD, ... ] | "FIELD, ... " | { has:PATTERN, nlp:PATTERN, bin:PATTERN, qex:PATTERN, browse:"FIELD,...", pivot: "FIELD,..." }

In addition, update journalling, search tracking, query broadcasting, and auto field conversion is 
supported using these ATTRIBUTES:

	unsafeok: 	[true] | false 		// allow potentially unsafe queries
	trace: [true] | false			// trace queries
	journal: true | [false] 		// enable table journalling
	search: "field,field,..." 		// define fulltext search fields
	track: true | [false] 		// enable search tracking
	ag: "..." 		// aggregate where/having with least(?,1), greatest(?,0), sum(?), ...

The select query will callback the CB = [each || all || clone || trace] handler with each/all record(s) matched 
by .where, indexed by  .index, ordered by .order ordering, grouped by .group, filtered by .having 
and limited by .limit ATTRIBUTES.  Select will search for PATTERN 
using its index.nlp (natural language parse), index.bin (binary mode), index.qex (query expansion), 
or group recording according to its index.browse (file navigation) or index.pivot (joint statistics).

Non-select queries will broadcast a change to all clients if a where.ID is presented (and an emiitter
was configured), and will journal the change when jounalling is enabled.

Example

### Access the neo4j database
	neoThread( neo => {	
		neo.cypher( "...", [ ... ], (err,recs) => {
		});
	});

Example

### Create dataset on a new sql thread
	sqlThread( sql => {
	
		var ds = new JSDB.DS(sql,{
			table:"test.x", 
			rec: (recs) => console.log(recs) 
		});
		
	});
	

Example

### Create dataset and access each record
	var ds = new JSDB.DS(sql,{
		table:"test.x",
		limit:[0,1],
		rec: function each(rec) {console.log(rec)}
	});
		
	var ds = new JSDB.DS(sql,{
		table:"test.x",
		trace:1,
		where:{ x: "x=12" },
		rec: function each(rec) {console.log(rec)}});
		
	var ds = new JSDB.DS(sql,{
		table:"test.x",
		trace:1,
		where:{ a: "a = 0.5"},
		rec: function each(rec) {console.log(rec)}
	});
	
	var ds = new JSDB.DS(sql,{
		table:"test.x",
		trace:1,
		where:{ a: "a<30"},
		rec: function each(rec) {console.log(rec)}
	});
	

Example

### Create dataset and access all records
	var ds = new JSDB.DS(sql,{
		table:"test.x",
		trace:1,
		where:{
			a: "a<30", 
			b: "b!=0",
			x: "x like '%find%'",
			ID: "ID=5"},
		rec: (recs) => console.log(recs)
	});
	
	var ds = new JSDB.DS(sql,{
		table:"test.x",
		trace:1,
		order:[{property:"a",direction:"asc"}],
		rec: (recs) => console.log(recs)
	});
	
	var ds = new JSDB.DS(sql,{
		table:"test.x",
		trace:1,
		index:{pivot:"root"},
		group:"a,b",
		rec: (recs) => console.log(recs)
	});
	

Example

### Select ds record(s) matched by ds.where
	ds.where = {ID: "ID=1"};
	ds.rec = (rec) => console.log(rec);
	

Example

### Delete ds record(s) matched by ds.where
	ds.where = {ID:"ID=2"}
	ds.rec = null
	

Example

### Update ds record(s) matched by ds.where
	ds.where = null
	ds.rec = [{a:1,b:2,ds:"hello"},{a:10,b:20,x:"there"}]
	ds.where = {ID: "ID=3"}
	ds.rec = {a:100} 

• JSDB
  • ~_Errors
    • .noConnect
    • .nillUpdate
    • .unsafeQuery
    • .unsupportedQuery
    • .invalidQuery
    • .noTable
    • .noDB
    • .noLock
    • .isUnlocked
    • .failLock
    • .isLocked
    • .noExe
    • .noRecord
  • ~_dsAttrs
  • ~_Dumpers
  • ~parseWild()
  • ~getContext()
  • ~getKeys()
  • ~getKeysFull()
  • ~getJsons()
  • ~getTexts()
  • ~getSearchables()
  • ~getGeometries()
  • ~getTables()
  • ~getDatabases()
  • ~context()
  • ~cache()
  • ~beginBulk()
  • ~endBulk()
  • ~flattenCatalog()
  • ~forFirst()
  • ~forEach()
  • ~forAll()
  • ~Index(ds, query, cb)
  • ~Delete(ds, where, cb)
  • ~Update(ds, where, body, cb)
  • ~Insert(ds, body, cb)
  • ~Select(ds, index, where, opts, cb)
  • ~ingestFile(path, opts, cb)
  • ~Serialize()
  • ~saveContext()
    • ~saveEvents(sql, evs, ctx, cb)
      • ~stashify(evs, watchKey, targetPrefix, ctx, stash, cb)
  • ~cypher()
  • ~clearNet()
  • ~saveNodes()
  • ~findAssoc()
  • ~saveNet()
  • ~saveEdges()

JSDB~_Errors

Client error messages

Kind: inner constant of JSDB

• ~_Errors
  • .noConnect
  • .nillUpdate
  • .unsafeQuery
  • .unsupportedQuery
  • .invalidQuery
  • .noTable
  • .noDB
  • .noLock
  • .isUnlocked
  • .failLock
  • .isLocked
  • .noExe
  • .noRecord

_Errors.noConnect

Kind: static property of _Errors

_Errors.nillUpdate

Kind: static property of _Errors

_Errors.unsafeQuery

Kind: static property of _Errors

_Errors.unsupportedQuery

Kind: static property of _Errors

_Errors.invalidQuery

Kind: static property of _Errors

_Errors.noTable

Kind: static property of _Errors

_Errors.noDB

Kind: static property of _Errors

_Errors.noLock

Kind: static property of _Errors

_Errors.isUnlocked

Kind: static property of _Errors

_Errors.failLock

Kind: static property of _Errors

_Errors.isLocked

Kind: static property of _Errors

_Errors.noExe

Kind: static property of _Errors

_Errors.noRecord

Kind: static property of _Errors

JSDB~_dsAttrs

Reserved for dataset attributes

Kind: inner constant of JSDB

JSDB~_Dumpers

Kind: inner constant of JSDB

JSDB~parseWild()

Kind: inner method of JSDB

JSDB~getContext()

Kind: inner method of JSDB

JSDB~getKeys()

Kind: inner method of JSDB

JSDB~getKeysFull()

Kind: inner method of JSDB

JSDB~getJsons()

Kind: inner method of JSDB

JSDB~getTexts()

Kind: inner method of JSDB

JSDB~getSearchables()

Kind: inner method of JSDB

JSDB~getGeometries()

Kind: inner method of JSDB

JSDB~getTables()

Kind: inner method of JSDB

JSDB~getDatabases()

Kind: inner method of JSDB

JSDB~context()

Kind: inner method of JSDB

JSDB~cache()

Implements generic cache. Looks for cache given opts.key and, if found, returns cached results on cb(results); otherwse, if not found, returns results via opts.make(probeSite, opts.parms, cb). If cacheing fails, then opts.default is returned. The returned results will always contain a results.ID for its cached ID. If a opts.default is not provided, then the cb callback in not made.

Kind: inner method of JSDB

JSDB~beginBulk()

Kind: inner method of JSDB

JSDB~endBulk()

Kind: inner method of JSDB

JSDB~flattenCatalog()

Flatten entire database for searching the catalog. Need to rework using Serialize

Kind: inner method of JSDB

JSDB~forFirst()

Kind: inner method of JSDB

JSDB~forEach()

Kind: inner method of JSDB

JSDB~forAll()

Kind: inner method of JSDB

JSDB~Index(ds, query, cb)

Index records.

Kind: inner method of JSDB

| Param | Type | Description | | --- | --- | --- | | ds | String | name of dataset table | | query | Object | hash of search options {"=": {....}, ">": {....}, ....} | | cb | function | callback(keys) |

JSDB~Delete(ds, where, cb)

Delete records.

Kind: inner method of JSDB

| Param | Type | Description | | --- | --- | --- | | ds | String | name of dataset table | | where | Object | hash of search options {"=": {....}, ">": {....}, ....} | | cb | function | callback(err,info) |

JSDB~Update(ds, where, body, cb)

Update records.

Kind: inner method of JSDB

| Param | Type | Description | | --- | --- | --- | | ds | String | name of dataset table | | where | Object | hash of search options {"=": {....}, ">": {....}, ....} | | body | Object | data for update | | cb | function | callback(err,info) |

JSDB~Insert(ds, body, cb)

Insert records.

Kind: inner method of JSDB

| Param | Type | Description | | --- | --- | --- | | ds | String | name of dataset table | | body | Object | data for insert | | cb | function | callback(err,info) |

JSDB~Select(ds, index, where, opts, cb)

Select records.

Kind: inner method of JSDB

| Param | Type | Description | | --- | --- | --- | | ds | String | name of dataset table | | index | Object | hash of { "TO":"FROM", "TO":"STORE$KEY", ....} keys to select | | where | Object | hash of search options {"=": {....}, ">": {....}, ....} | | opts | Object | limit,offset,client,pivot,browse,sort options | | cb | function | callback(err,recs) |

JSDB~ingestFile(path, opts, cb)

Ingest a comma-delimited, column-headered stream at path using the supplied streaming options. Records are inserted into the sql target table defined by the path = /.../target.type. The keys="recKey:asKey sqlType,..." defines how record values are stored.

Kind: inner method of JSDB

| Param | Type | Description | | --- | --- | --- | | path | String | source file | | opts | Object | {keys,comma,newline,limit,as,batch} streaming options | | cb | function | Callback([record,...]) |

JSDB~Serialize()

Serialize a select query.

Kind: inner method of JSDB
Example

sql.Serialize({
		ds1: "SELECT ... ",
		ds2: "SELECT ... ", ...
		ds3: "/dataset?...", 
		ds4: "/dataset?...", ...
	}, ctx, ctx => {
		// ctx[ ds1 || ds2 || ... ] records
	});

JSDB~saveContext()

Aggregate and save events evs = [ev, ...] || { } under direction of the supplied context ctx = { Save: { ... }, Ingest: true||false, Export: true||false, ... }. Stashify is used to aggreagate data using [ev, ...].stashify( "at", "Save_", ctx ) where events ev = { at: KEY, A: a1, B: b1, ... } || { x: x1, y: y1 } are saved in Save_KEY = {A: [a1, a2, ...], B: [b1, b2, ...], ...} iff Save_KEY is in the supplied ctx.

Kind: inner method of JSDB

• ~saveContext()
  • ~saveEvents(sql, evs, ctx, cb)
    • ~stashify(evs, watchKey, targetPrefix, ctx, stash, cb)

saveContext~saveEvents(sql, evs, ctx, cb)

Stash aggregated events evs = { at: "AT", ... } into context Save_AT keys then callback cb with remaining events.

Kind: inner method of saveContext

| Param | Type | Description | | --- | --- | --- | | sql | object | sql connection | | evs | object | events to be saved | | ctx | object | notebook context | | cb | function | callback(ev,stat) |

saveEvents~stashify(evs, watchKey, targetPrefix, ctx, stash, cb)

Aggregate ctx keys into optional Save_KEY stashes such that:

[	
	{ at: "KEY", A: a1, B: b1, ... }, 
	{ at: "KEY", A: a2, B: b2, ... }, ... 
	{ x: x1, y: y1 },
	{ x: x2, y: y2 },	...
].stashify( "at", "Save_", {Save_KEY: {}, ...} , stash, cb )

creates stash.Save_KEY = {A: [a1, a2, ...], B: [b1, b2, ...], ...} iff Save_KEY is in the supplied context ctx. If no stash.remainder is provided by the ctx, the {x, y, ...} are appended (w/o aggregation) to stash.remainder. Conversely, if ctx contains a stash.remainder, the {x, y, ...} are aggregated to stash.remainder.

Kind: inner method of saveEvents

| Param | Type | Description | | --- | --- | --- | | evs | object | events to be saved | | watchKey | String | this = [ { watchKey:"KEY", x:X, y: Y, ...}, ... } | | targetPrefix | String | stash = { (targetPrefix + watchKey): { x: [X,...], y: [Y,...], ... }, ... } | | ctx | Object | plugin context keys | | stash | Object | refactored output suitable for a Save_KEY | | cb | function | callback(ev,stat) returns refactored result to put into stash |

JSDB~cypher()

Kind: inner method of JSDB

JSDB~clearNet()

Kind: inner method of JSDB

JSDB~saveNodes()

Kind: inner method of JSDB

JSDB~findAssoc()

Kind: inner method of JSDB

JSDB~saveNet()

Kind: inner method of JSDB

JSDB~saveEdges()

Kind: inner method of JSDB

Contacting, Contributing, Following

Feel free to

• submit and status TOTEM issues
• contribute to TOTEM notebooks
• revise TOTEM requirements
• browse TOTEM holdings
• or follow TOTEM milestones

License

MIT

© 2012 ACMESDS