@totemorg/agent

v1.14.0

Published

6 months ago

Totem agent

Downloads

0High
0Medium
0Low

AGENT

Provides methods to attach agents, to regulate jobs, and to fetch data.

Install

npm install @totemorg/agent	
npm update

Start

npm run start 					# Start/unit-test

Manage

npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation
npm run pubminor				# republish as minor version
npm run pubmajor				# republish as major version

Usage

Acquire and optionally configure AGENT as follows:

import { $ } from "@totemorg/agent");

See Program Reference for usage details.

Program Reference

agent

Attach agents to endpoints. Regulate job workflows. Fetch data, file, graph, and json stores. This module documented IAW jsdoc.

From a remote agent node (a @totemorg/agent-deprived node), get/revise the agent registration template, or get/revise a agent service template from your local @totemorg/totem to register agents with the agent cloud.

From a native agent nodes (a @totemorg/agent-ready node), agents directly attach themselves to the agent cloud using the agent attach template.

Requires: module:http, module:https, module:os, module:vm, module:enums

agent
- static
- inner

agent.attachAgents(opts, agents, cb) ⇒

Attach agents to the agent cloud, where each agent(req,res) specifies its endpoint as follows:

NAME: agent                     // handle notebook/data requests
"/AREA": agent                  // handle file requests
".TYPE": agent                  // handle conversion requests 
"/": agent 	                    // default handler for file requests 
".": agent	                    // default handler for type conversions
NAME_MACHINE: agent             // handle requests via a MACHINE = py|cv|m|jx|db| ... 
_IGNORE: agent                  // ignored
"#EVENT": agent                 // handle notebook EVENT = charge|run requests
GET|PUT|POST|DELETE: agent      // handle CRUD data requests

The supplied agents hash is automatically extended with "/json", "/graph", and "/file" reader agents.

When a request arrives to a agent server , attachAgents first

attempts to resolve the request to the master agent, then (if SQL-equipped)
attempts to resolve the request to a remote agent, then
attempts to resolve the request to a notebook agent, then
defaults to a CRUD agent based on the request type.

The following options opts can be provided when attaching agents:

cores number of servers (workers) attached to the agents.
port number that servers listen on.
CORS flag to support Cross-Origin-Scripting.
multipart flag to permit multipart files in request bodies.
never = file cacheing exemption hash { AREA: 1, PATH: 1, ... }.
mimes mime hash sets the mime header for requested TYPE.
https options hash when creating a HTTPS server.
map = area remapping hash {AREA: "/MappedPath", ...}.
access(req) method to validate client request req.
proxy(cert,headers) method to adjust cert from http headers.
init(server,agents) method to initialize the agent server after listener attached.
busy() method to block DOS attacks. Machines* require additional modules:
py,cv,m requires @totemorg/agent
jx requires @totemorg/man
db requires @totemorg/enums

to support installed python, opencv, matlab, mathjx, mysql db, and neo4j db.

Each agent(req,res) is provided a res response method accepting a:

string
object
array
error
function

and a req request containing:

cookie: "...."		// client cookie string
agent: "..."		// client browser info
ipAddress: "..."	// client ip address
referer: "http://site"		// url during a cross-site request
method: "GET|PUT|POST|DELETE" 		// http request method
now: date			// date stamp when request started
post: "..."			// raw body text
url	: "/query"		// requested url path
reqSocket: socket	// socket to retrieve client cert, post etc
resSocket: socket	// method to create socket to accept response
cert: {...} 		// full client cert

path: "/[area/...]name.type"	// full node path
area: "name"		// file area being requested
table: "name"		// name of dataset/table being requested
type: "type" 		// type descriptor 

query: {...} 		// raw keys from url
where: {...} 		// sql-ized query keys from url
body: {...}			// body keys from request 
flags: {...} 		// flag keys from url
index: {...}		// sql-ized index keys from url

To attach your nodejs-ready machine to the agent cloud:

curl "http://totem/attach" | node [- ?|option=value|list|file.js]

where option overrides the default port,cores,agents,etc.

Kind: static method of agent
Returns: macs Context of machines required by the agents

| Param | Type | Description | | --- | --- | --- | | opts | Object | Service options | | agents | function | Object | Hash of (req,res)-agents | | cb | function | Optional callback(agent) to process resolved agent |

.attachAgents(opts, agents, cb) ⇒

attachAgents~getGraph(req, res)

Graph reader agent returns neo4j graph at the requested path = /AGENT/GRAPH?QUERY

Kind: inner method of attachAgents

| Param | Type | Description | | --- | --- | --- | | req | Object | totem session request | | res | function | totem session response |

attachAgents~getJson(req, res)

Json reader Agent returns json store at the requested path = /AGENT/TABLE/STORE$EXPRESSION.

Kind: inner method of attachAgents

| Param | Type | Description | | --- | --- | --- | | req | Object | totem session request | | res | function | totem session response |

attachAgents~getFile(req, res)

File reader agent returns file at the requested path = /AREA/FILE

Kind: inner method of attachAgents

| Param | Type | Description | | --- | --- | --- | | req | Object | totem session request | | res | function | totem session response |

attachAgents~getAgent(spec, agents)

Return agent from the agents hash with the given {type,area,method,table} spec.

Kind: inner method of attachAgents

| Param | Type | Description | | --- | --- | --- | | spec | Object | {type,area,method,table} to derive agent | | agents | Object | hash of agents |

agent.regulateJob(spec, ctx, cb)

Regulate callback to cb(spec,ctx) on events defined by an event spec:

spec = "STEP / ONSTEPS / OFFSTEPS / START / END" || [spec, ...]

where each subspec is of the form

subspec = NUMBER | NUMBER [s|m|h|d|y] | dd-mm-yr

The context ctx can define:

ctx.queue = "task queue" to share job status
ctx.batch = NUMBER of records returned in callback cb(t,recs)
ctx.client = "client name" to chage this request
ctx.source = "sql table" record source
ctx.fields = "field, ..." source fields
ctx.priority = NUMBER of seconds to monitor signoff-status if required by client

Kind: static method of agent

| Param | Type | Description | | --- | --- | --- | | spec | String | Job regulation spec "STEP / ONSTEPS / OFFSTEPS / START / END" || [spec, ...] | | ctx | Object | Job context | | cb | function | Job callback(t,ctx) |

agent.fetchData(ref, data, ack)

Fetch data from a specified ref url

PROTOCOL://HOST/FILE ? QUERY

where PROTOCOL is

http(s) = http (https) protocol
curl(s) = curl (curls uses certs/fetch.pfx to authenticate)
wget(s) = wget (wgets uses certs/fetch.pfx to authenticate)
mask = http access via rotated proxies
lexnex = Lexis-Nexis oauth access to documents
mysql = mysql database
neo4j = neo4j database
csv = csv file
stream = stream file
json = json file
jpg = image file
list = list file
null = null file
file = file/folder
SITEREF = shortcut name

File paths may also contain wild-* cards. The callback name (GET || PUT || POST || DELETE || JSON) determines the request method. Optional QUERY keys:

batch 	= NUMBER of records to fetch 
offset	= NUMBER of records to offset fetch
select	="FIELD, ..." to fetch
keys	= [...] header keys in csv source
comma	= "delim" for csv source
newline = "delim" for csv source

Kind: static method of agent

| Param | Type | Description | | --- | --- | --- | | ref | String | source URL | | data | function | object | or ack callback(results||null) | | ack | function | callback(results||null) |

Example

fetchData( ref, text => { // get request
})

Example

fetchData( ref, function json(ctx) {    // get request with json ctx
})

Example

fetchData( ref, [ ... ], function post(stat) { 	// post request with data hash list
})

Example

fetchData( ref, { ... }, function put(stat) { 	// put request with data hash
})

Example

fetchData( ref, null, function deete(stat) {	// delete request 
})

agent.LexisNexis()

Start N Lexis-Nexis scrapes residing at the endpoint endpt + random counter when R is non-zero.

Kind: static method of agent

agent~quickLinks

Fetch quick SITEREFs

    https://www.programmableweb.com/search/military

    ACLED
    https://www.programmableweb.com/api/acled-rest-api-v25
    ACCT [email protected] / ACLEDsnivel1
    API https://developer.acleddata.com/rehd/cms/views/acled_api/documents/API-User-Guide.pdf
    SITE https://developer.acleddata.com/
    The Armed Conflict Location & Event Data Project (ACLED) is a real-time data and and crisis analysis and mapping project on political violence and protest across the world. ACLED's mission is to produce dis-aggregated, locally informed data and analysis globally in real time. An ACLED REST API enables users to retrieve data about actors, actor type, country, region and get data in JSON, XML, CSV or text. Filter data by date, area, pagination, etc.

    Animetrics FIMS
    https://www.programmableweb.com/api/animetrics-fims-cloud-rest-api
    http://animetrics.com/fims-cloud
    Aimed at the law enforcement, security and military intelligence industries, Animetrics' FaceR Identity Management Solution (FIMS) allows organizations to bring mobile security and video surveillance facial-biometrics applications into the field for use in real time. FIMS Cloud is used to centralize and access a user's cloud based photographic stores using facial recognition. FIMS Cloud organizes, searches and centralizes access to photographic stores using 1:many web service based verification engine. Access to the service is provided via a RESTful API. Public documentation is not available.

    Navlost WXT Weather Tesseract
    https://www.programmableweb.com/api/navlost-wxt-weather-tesseract
    The WXT Weather Service provides atmospheric weather information through a REST architecture, HTTP requests, and JSON formats. It integrates METAR/TAF information, sun, and moon calculations, targeting aviation and energy applications. Advanced features include: -Upper atmosphere information (e.g., research, aviation, rocketry, military) -Automated, push-type notification of arbitrary weather-related events (alert service) -Calculation of arbitrary results derived from weather forecast information via a server-side scripting language. The default response type is application/json, although other formats may be supported. At the present time, there is partial support for comma-separated value (CSV) responses.
    https://wxt.navlost.eu/api/v1/
    https://wxt.navlost.eu/doc/api/

Kind: inner constant of agent

agent~maxRetry

Fetch wget/curl maxRetry

Kind: inner constant of agent

agent~certs

Fetch certs

Kind: inner constant of agent

Contacting, Contributing, Following

Feel free to

submit and status TOTEM issues
contribute to TOTEM notebooks
revise TOTEM requirements
browse TOTEM holdings
or follow TOTEM milestones

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

AGENT

Install

Start

Manage

Usage

Program Reference

agent

agent.attachAgents(opts, agents, cb) ⇒

attachAgents~getGraph(req, res)

attachAgents~getJson(req, res)

attachAgents~getFile(req, res)

attachAgents~getAgent(spec, agents)

agent.regulateJob(spec, ctx, cb)

agent.fetchData(ref, data, ack)

agent.LexisNexis()

agent~quickLinks

agent~maxRetry

agent~certs

Contacting, Contributing, Following