@totemorg/totem
v3.15.0
Published
Provides basic, customizable and extensible web service
Downloads
74
Readme
TOTEM
TOTEM provides a barebones web service with customizable features:
- attach agents (local, registered, and notebook) to process endpoint requests
- configure https service
- denial-of-service protection
- secure connections
- client logins and profiles
- PKI encryption and authentication
- fault protected run states
- indexing, uploading, downloading and cacheing static files
- crud interface to mysql and neo4j databases
- task queuing and regulation
- periodic watchdog and polling services
- file streams
- data fetching using various protocols
- smartcard reader
- site skinning
TOTEM provides the following:
/DATASET.TYPE ? QUERY
/AREA/FILE ? QUERY
/AREA/ ? QUERY
/json/STORE ? QUERY
/graph/GRAPH ? QUERY
/AGENT ? QUERY
where the optional TYPE:
db | csv | txt | html | json
returns the dataset in the specified form.
Install
npm install @totemorg/totem # install base service
npm update
Start
npm run ?env # Show env variables
npm run ?config # show configuration
npm run ? # List start options
npm run start PLUGIN PLUGIN... # Start totem with optional plugins
npm run setprot # Configure for protected mode
npm run setdebug # Configure for debugging mode
npm run setoper # Configure for operational mode
npm run setprod # Configure for production mode
npm run startdbs # Start database servers
Manage
npm run verminor # Roll minor version
npm run vermajor # Roll major version
npm run redoc # Regen documentation
npm run pubminor # republish as minor version
npm run pubmajor # republish as major version
Usage
import * as TOTEM from "@totemorg/totem";
TOTEM.config({
key: value, // set key
"key.key": value, // indexed set
"key.key.": value // indexed append
}, sql => {
console.log( sql ? "look mom - Im running!" : "something evil is lurking" );
});
where configuration keys follow ENUMS deep copy conventions.
Program Reference
TOTEM
Provides a basic web service. This module documented IAW jsdoc.
Requires: module:enums, module:jsdb, module:securelink, module:skin, module:agent, module:fs, module:constants, module:cluster, module:child_process, module:stream, module:vm, module:crypto, module:mime, module:xml2js, module:toobusy-js, module:json2csv, module:js2xmlparser, module:cheerio, module:markdown
Author: ACMESDS
Example
// npm test T1
// Create simple service but dont start it.
Log({
msg: "Im simply a Totem interface so Im not even running as a service",
default_fetcher_endpts: TOTEM.nodeRouters,
default_protect_mode: TOTEM.guard,
default_cores_used: TOTEM.cores
});
Example
// npm test T2
// Totem service running in fault protection mode, no database, no UI; but I am running
// with 2 workers and the default endpoint routes.
config({
mysql: null,
guard: true,
cores: 2
}, sql => {
Log(
`I'm a Totem service running in fault protection mode, no database, no UI; but I am running
with 2 workers and the default endpoint routes` );
});
Example
// npm test T3
// A Totem service with no workers.
config({
}, sql => {
Log(
`I'm a Totem service with no workers. I do, however, have a mysql database from which I've derived
my startup options (see the openv.apps table for the Nick="Totem1") with just the standard endpoints.
`
);
});
Example
// npm test T4
// Only 1 worker, unprotected, a mysql database, and two endpoints.
config({
nodeRouters: {
dothis: function dothis(req,res) { //< named handlers are shown in trace in console
res( "123" );
Log("", {
do_query: req.query
});
},
dothat: function dothat(req,res) {
if (req.query.x)
res( [{x:req.query.x+1,y:req.query.x+2}] );
else
res( _Error("We have a problem huston") );
Log("", {
msg: `Like dothis, but needs an ?x=value query`,
or_query: req.query,
or_user: req.client
});
}
}
}, sql => {
Log("", {
msg:
`As always, if the openv.apps Encrypt is set for the Nick="Totem" app, this service is now **encrypted** [*]
and has https (vs http) endpoints, here /dothis and /dothat endpoints. Ive only requested only 1 worker (
aka core), Im running unprotected, and have a mysql database.
[*] If my NICK.pfx does not already exists, Totem will create its password protected NICK.pfx cert from the
associated public NICK.crt and private NICK.key certs it creates.`,
my_endpoints: T.nodeRouters
});
});
Example
// npm test T5
// no cores but a mysql database and an anti-bot shield
config({
"login.challenge.extend": 20
}, sql => {
Log("", {
msg:
`I am Totem client, with no cores but I do have mysql database and I have an anti-bot shield!! Anti-bot
shields require a Encrypted service, and a UI (like that provided by DEBE) to be of any use.`,
mysql_derived_parms: T.siteContext
});
});
Example
// npm test T6
// Testing tasker with database, 3 cores and an additional /test endpoint.
config({
guard: false, // ex override default
cores: 3, // ex override default
"nodeRouters.": { // define endpoints
test: function (req,res) {
res(" here we go"); // endpoint must always repond to its client
if (isMaster) // setup tasking examples on on master
switch (req.query.opt || 1) { // test example runTask
case 1:
T.runTask({ // setup tasking for loops over these keys
keys: "i,j",
i: [1,2,3],
j: [4,5]
},
// define the task which returns a message msg
($) => "hello i,j=" + [i,j] + " from worker " + _Worker + " on " + $.node,
// define the message msg handler
(msg) => console.log(msg)
);
break;
case 2:
T.runTask({
qos: 1,
keys: "i,j",
i: [1,2,3],
j: [4,5]
},
($) => "hello i,j=" + [i,j] + " from worker " + _Worker + " on " + $.node,
(msg) => console.log(msg)
);
break;
case 3:
break;
}
}
}
}, sql => {
Log( "Testing runTask with database and 3 cores at /test endpoint" );
});
Example
// npm test T7
// Conduct db maintenance
config({
}, sql => {
Log( "db maintenance" );
if (isMaster)
switch (process.argv[3]) {
case 1:
sql.query( "select voxels.id as voxelID, chips.id as chipID from openv.voxels left join openv.chips on voxels.Ring = chips.Ring", function (err,recs) {
recs.forEach( rec => {
sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
Log(err);
});
});
});
break;
case 2:
sql.query("select ID, Ring from openv.voxels", function (err, recs) {
recs.forEach( rec => {
sql.query(
"update openv.voxels set Point=geomFromText(?) where ?",
[ `POINT(${rec.Ring[0][0].x} ${rec.Ring[0][0].y})` , {ID: rec.ID} ],
err => {
Log(err);
});
});
});
break;
case 3:
sql.query( "select voxels.id as voxelID, cache.id as chipID from openv.voxels left join openv.cache on voxels.Ring = fileCache.geo1", function (err,recs) {
Log(err);
recs.forEach( rec => {
sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
Log(err);
});
});
});
break;
case 4:
sql.query("select ID, geo1 from openv.cache where bank='chip'", function (err, recs) {
recs.forEach( rec => {
if (rec.geo1)
sql.query(
"update openv.cache set x1=?, x2=? where ?",
[ rec.geo1[0][0].x, rec.geo1[0][0].y, {ID: rec.ID} ],
err => {
Log(err);
});
});
});
break;
case 5:
var parms = {
ring: "[degs] closed ring [lon, lon], ... ] specifying an area of interest on the earth's surface",
"chip length": "[m] length of chip across an edge",
"chip samples": "[pixels] number of pixels across edge of chip"
};
//get all tables and revise field comments with info data here - archive parms - /parms in flex will
//use getfileds to get comments and return into
case 6:
var
RAN from "../randpr"),
ran = new RAN({
models: ["sinc"],
Mmax: 150, // max coherence intervals
Mstep: 5 // step intervals
});
ran.config( function (pc) {
var
vals = pc.values,
vecs = pc.vectors,
N = vals.length,
ref = vals[N-1];
vals.forEach( (val, idx) => {
var
save = {
correlation_model: pc.model,
coherence_intervals: pc.intervals,
eigen_value: val,
eigen_index: idx,
ref_value: ref,
max_intervals: ran.Mmax,
eigen_vector: JSON.stringify( vecs[idx] )
};
sql.query("INSERT INTO openv.pcs SET ? ON DUPLICATE KEY UPDATE ?", [save,save] );
});
});
break;
}
});
Example
// npm test T8
// Conduct neo4j database maintenance
const $ from "../man/man.js");
config();
neoThread( neo => {
neo.cypher( "MATCH (n:gtd) RETURN n", {}, (err,nodes) => {
Log("nodes",err,nodes.length,nodes[0]);
var map = {};
nodes.forEach( (node,idx) => map[node.n.name] = idx );
//Log(">map",map);
neo.cypher( "MATCH (a:gtd)-[r]->(b:gtd) RETURN r", {}, (err,edges) => {
Log("edges",err,edges.length,edges[0]);
var
N = nodes.length,
cap = $([N,N], (u,v,C) => C[u][v] = 0 ),
lambda = $([N,N], (u,v,L) => L[u][v] = 0),
lamlist = $(N, (n,L) => L[n] = [] );
edges.forEach( edge => cap[map[edge.r.srcId]][map[edge.r.tarId]] = 1 );
//Log(">cap",cap);
for (var s=0; s<N; s++)
for (var t=s+1; t<N; t++) {
var
{cutset} = $.MaxFlowMinCut(cap,s,t),
cut = lambda[s][t] = lambda[t][s] = cutset.length;
lamlist[cut].push([s,t]);
}
lamlist.forEach( (list,r) => {
if ( r && list.length ) Log(r,list);
});
});
});
});
- TOTEM
TOTEM.openvACL
Route nodes according to security requirements.
Kind: static constant of TOTEM
TOTEM.filters
Endpoint filters cb(data data as string || error)
Kind: static constant of TOTEM
Cfg: Object
TOTEM.watchDogs
Kind: static constant of TOTEM
TOTEM.siteContext
Site context extended by the mysql derived query when service starts
Kind: static constant of TOTEM
Cfg: Object
TOTEM.uploadFile(client, srcStream, sinkPath, tags, cb)
Uploads a source stream srcStream
to a target file sinkPath
owned by the
specified client
; optional tags
are tagged to the upload and the callback
cb
is made if the upload was successful.
Kind: static method of TOTEM
| Param | Type | Description | | --- | --- | --- | | client | String | file owner | | srcStream | Stream | source stream | | sinkPath | String | path to target file | | tags | Object | hash of tags to add to file | | cb | function | callback(file) if upload successful |
TOTEM~agents : object
Endpoint agents each of the form agent(req,res)
Kind: inner namespace of TOTEM
- ~agents : object
agents.attach(req, res)
Register named agents
at the specified port
serviced by specified number of server cores
, charging clients
a specified usage cost
, or collect util
info from registered agents. See attachAgents for details.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.build(req, res)
Endpoint to build script for specified &install
keys.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.favicon(req, res)
Endpoint to return favorite icon
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.help(req, res)
Endpoint to return system help on requested agent
and section goto
.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.ping(req, res)
Endpoint to test connectivity.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | Object | Totem request | | res | function | Totem response |
agents.GET(req, res)
Endpoint to select records from a dataset or redirect to an agent.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.PUT(req, res)
Endpoint to update records in a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.DELETE(req, res)
Endpoint to delte records from a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.POST(req, res)
Endpoint to add records to a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.EXEC(req, res)
Endpoint to execute records in a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
TOTEM~_Errors
Client error messages
Kind: inner constant of TOTEM
_Errors.noFile
Kind: static property of _Errors
_Errors.badFilter
Kind: static property of _Errors
_Errors.noData
Kind: static property of _Errors
_Errors.noAgent
Kind: static property of _Errors
_Errors.noDB
Kind: static property of _Errors
_Errors.badReturn
Kind: static property of _Errors
_Errors.badLogin
Kind: static property of _Errors
_Errors.noSocket
Kind: static property of _Errors
_Errors.badQuery
Kind: static property of _Errors
_Errors.noEndpoint
Kind: static property of _Errors
_Errors.noID
Kind: static property of _Errors
_Errors.noPost
Kind: static property of _Errors
_Errors.noSQL
Kind: static property of _Errors
_Errors.noNEO
Kind: static property of _Errors
_Errors.noReader
Kind: static property of _Errors
TOTEM~agentThread(agent, req, res)
Start an agent thread by Route NODE = /DATASET.TYPE requests using the configured agents.
The routeThread adds client data:
encrypted: bool // true if request on encrypted server
site: {...} // site context
mimi: "type" // mime type of response
Kind: inner method of TOTEM
| Param | Type | Description | | --- | --- | --- | | agent | function | agent to run | | req | Object | session request | | res | responseCallback | session response |
TOTEM~dataThread(req, cb)
Start a dataset thread by appending a
sql: {...} // sql connector
ds: "focus.table" // fully qualified sql table
focus: user focus
//action: "select|update| ..." // corresponding crude name
to the req
request with callback cb(req).
Kind: inner method of TOTEM
| Param | Type | Description | | --- | --- | --- | | req | Object | Totem endpoint request | | cb | function | callback(revised req) |
TOTEM~startDogs()
Start watchdogs
Kind: inner method of TOTEM
TOTEM~initServer()
Initialize the service.
Kind: inner method of TOTEM
TOTEM~stopServer()
Stop the server.
Kind: inner method of TOTEM
String
- String
- ~linkify(ref) ⇐ String
- ~mailify(ref) ⇐ String
- ~parseXML(cb) ⇐ String
String~linkify(ref) ⇐ String
Expands "*"-tagged string into html link.
Kind: inner method of String
Extends: String
| Param | Type | | --- | --- | | ref | String |
Example
"test [abc](http:/junk)".linkify()
==> 'test <a href="http:/junk">abc</a>'
Example
"here is a link".linkify("test")
==> '<a href="test">here is a link</a>'
String~mailify(ref) ⇐ String
Expands string into a html email link.
Kind: inner method of String
Extends: String
| Param | Type | | --- | --- | | ref | String |
Example
"contact me".mailify("user.one")
'<a href="mailto:contact me">user.one</a>'
String~parseXML(cb) ⇐ String
Parse XML string into json then callsback cb(json).
Kind: inner method of String
Extends: String
| Param | Type | Description | | --- | --- | --- | | cb | function | callback( json || null if error ) |
Array
Array~groupify(dot)
Groups [ "x.y.z", "u,v", "x" , ... ] list entries into "x(y(z)), u(v), x, ..." string.
Kind: inner method of Array
| Param | Type | Description | | --- | --- | --- | | dot | string | item seperator |
Example
["a.b","x.y.z"].groupify(".")
==> 'a(b),x(y(z))'
Array~blog(req, keys, cb)
Blogs each string in the list.
Kind: inner method of Array
See: totem:blogify
| Param | Type | Description | | --- | --- | --- | | req | Object | Request context | | keys | List | list of keys to blog | | cb | function | callback(recs) blogified version of records |
Array~joinify(cb)
Joins a list with an optional callback cb(head,list) to join the current list with the current head.
Kind: inner method of Array
| Param | Type | | --- | --- | | cb | function |
Example
[ a: null,
g1: [ b: null, c: null, g2: [ x: null ] ],
g3: [ y: null ] ].joinify()
==>
"a,g1(b,c,g2(x)),g3(y)"
TOTEM
- TOTEM
TOTEM.openvACL
Route nodes according to security requirements.
Kind: static constant of TOTEM
TOTEM.filters
Endpoint filters cb(data data as string || error)
Kind: static constant of TOTEM
Cfg: Object
TOTEM.watchDogs
Kind: static constant of TOTEM
TOTEM.siteContext
Site context extended by the mysql derived query when service starts
Kind: static constant of TOTEM
Cfg: Object
TOTEM.uploadFile(client, srcStream, sinkPath, tags, cb)
Uploads a source stream srcStream
to a target file sinkPath
owned by the
specified client
; optional tags
are tagged to the upload and the callback
cb
is made if the upload was successful.
Kind: static method of TOTEM
| Param | Type | Description | | --- | --- | --- | | client | String | file owner | | srcStream | Stream | source stream | | sinkPath | String | path to target file | | tags | Object | hash of tags to add to file | | cb | function | callback(file) if upload successful |
TOTEM~agents : object
Endpoint agents each of the form agent(req,res)
Kind: inner namespace of TOTEM
- ~agents : object
agents.attach(req, res)
Register named agents
at the specified port
serviced by specified number of server cores
, charging clients
a specified usage cost
, or collect util
info from registered agents. See attachAgents for details.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.build(req, res)
Endpoint to build script for specified &install
keys.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.favicon(req, res)
Endpoint to return favorite icon
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.help(req, res)
Endpoint to return system help on requested agent
and section goto
.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.ping(req, res)
Endpoint to test connectivity.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | Object | Totem request | | res | function | Totem response |
agents.GET(req, res)
Endpoint to select records from a dataset or redirect to an agent.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.PUT(req, res)
Endpoint to update records in a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.DELETE(req, res)
Endpoint to delte records from a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.POST(req, res)
Endpoint to add records to a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
agents.EXEC(req, res)
Endpoint to execute records in a dataset within the client's focus.
Kind: static method of agents
| Param | Type | Description | | --- | --- | --- | | req | requestHash | session request | | res | responseCallback | session response |
TOTEM~_Errors
Client error messages
Kind: inner constant of TOTEM
_Errors.noFile
Kind: static property of _Errors
_Errors.badFilter
Kind: static property of _Errors
_Errors.noData
Kind: static property of _Errors
_Errors.noAgent
Kind: static property of _Errors
_Errors.noDB
Kind: static property of _Errors
_Errors.badReturn
Kind: static property of _Errors
_Errors.badLogin
Kind: static property of _Errors
_Errors.noSocket
Kind: static property of _Errors
_Errors.badQuery
Kind: static property of _Errors
_Errors.noEndpoint
Kind: static property of _Errors
_Errors.noID
Kind: static property of _Errors
_Errors.noPost
Kind: static property of _Errors
_Errors.noSQL
Kind: static property of _Errors
_Errors.noNEO
Kind: static property of _Errors
_Errors.noReader
Kind: static property of _Errors
TOTEM~agentThread(agent, req, res)
Start an agent thread by Route NODE = /DATASET.TYPE requests using the configured agents.
The routeThread adds client data:
encrypted: bool // true if request on encrypted server
site: {...} // site context
mimi: "type" // mime type of response
Kind: inner method of TOTEM
| Param | Type | Description | | --- | --- | --- | | agent | function | agent to run | | req | Object | session request | | res | responseCallback | session response |
TOTEM~dataThread(req, cb)
Start a dataset thread by appending a
sql: {...} // sql connector
ds: "focus.table" // fully qualified sql table
focus: user focus
//action: "select|update| ..." // corresponding crude name
to the req
request with callback cb(req).
Kind: inner method of TOTEM
| Param | Type | Description | | --- | --- | --- | | req | Object | Totem endpoint request | | cb | function | callback(revised req) |
TOTEM~startDogs()
Start watchdogs
Kind: inner method of TOTEM
TOTEM~initServer()
Initialize the service.
Kind: inner method of TOTEM
TOTEM~stopServer()
Stop the server.
Kind: inner method of TOTEM
Contacting, Contributing, Following
Feel free to
- submit and status TOTEM issues
- contribute to TOTEM notebooks
- revise TOTEM requirements
- browse TOTEM holdings
- or follow TOTEM milestones
License
© 2012 ACMESDS