npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@totemstan/atomic

v1.8.0

Published

Cloud compute for python, js, cv, matlab, R, etc engines

Downloads

8

Readme

ATOMIC

ATOMIC provides cloud computing for python, js, cv, matlab, R, ... engines via the following web endpoints:

POST advance/step/insert a stateful engine
PUT	compile/init/update a stateful engine
DELETE deallocate/kill/delete a stateful engine
GET execute/read/select a stateless engines

Stateless engines are supported at the read (GET) endpoint, and are provided the following parameters:

TAU.i = {tau} = input event sinked to an engine
TAU.o = {tau} = output event sourced from an engine
TAU.p = {sql: {...}, query: {...} }

where the query hash will contain the supplied url parameters.

Stateful engines implement the step-init-kill (POST-PUT-DELETE) endpoints, and are supplied event tokens (tau):

TAU.i = [{tau}, ...] = events arriving to engine's input port
TAU.o = [{tau}, ...] = events departing from engine's output port
TAU.p = {port1: {...}, ... port2: {...}, ... sql: {...} }
TAU.port = engine's in/out port to step
TAU.thread = engine's 0-base thread counter

where input/output ports and engine code are defined by the engine context || COE || SBU.

An event token typically contain the following default fields (they can be freely interpretted and extended by the engine):

job = "" 	= Current job thread N.N...
work = 0 	= Anticipated/delivered data volume (dims bits etc)
disem = "" 	= Disemination channel for this event
classif = ""	= Classification of this event
cost = ""	= Billing center
policy = ""	= Data retention policy
status = 0	= Status code
value = 0	= Flow calculation

Manage

npm install @totemstan/atomic	# install
npm run start [ ? | $ | ...]	# Unit test
npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation

Usage

Acquire and optionally configure ATOMIC like this:

var ATOMIC = require("atomic").config({
	key: value, 						// set key
	"key.key": value, 					// indexed set
	"key.key.": value					// indexed append
});

where configuration keys follow ENUMS deep copy conventions

Program Reference

ATOMIC

Provides cloud computing on python, js, cv, matlab, R, ... engines via web endpoints. This module documented in accordance with jsdoc.

Requires: module:enums, module:child_process, module:fs, module:vm, module:pythonIF, module:opencvIF, module:RIF
Author: ACMESDS

Env Dependencies

HASGPU = 1|0
HASCAFFE = 1|0  

Example

### Totem and Atomic Engine interfaces:

	var ATOMIC = require("../atomic");
	var TOTEM = require("../totem");

	Trace( "A Totem+Engine client has been created", {
		a_tau_template: ATOMIC.tau("somejob.pdf"),
		engine_errors: ATOMIC.error,
		get_endpts: TOTEM.reader,
		my_paths: TOTEM.paths
	});

Example

### Totem being powered up and down:

	var TOTEM = require("../totem");

	TOTEM.config({}, function (err) {
		Trace( err || "Started but I will now power down" );
		TOTEM.stop();
	});

	var ATOMIC = require("../engine").config({
		thread: TOTEM.thread
	});

Example

### Totem service with a chipper engine endpoint and a database:

	var TOTEM = require("../totem").config({
		"byType.": {
			chipper: function Chipper(req,res) {				
				res( 123 );
			}
		},

		mysql: {
			host: ENV.MYSQL_HOST,
			user: ENV.MYSQL_USER,
			pass: ENV.MYSQL_PASS
		}

	});

	var ATOMIC = require("../engine").config({
		thread: TOTEM.thread
	});

Example

### Totem with a complete engine test endpoint:

	var TOTEM = require("../totem").config({
		"byType.": {
			test: function Chipper(req,res) {

				var itau = [ATOMIC.tau()];
				var otau = [ATOMIC.tau()];

				switch (req.query.config) {
					case "cv": // program and step haar opencv machine 
						parm =	{
							tau: [], 
							ports: {
								frame:	 {},
								helipads: {scale:0.05,dim:100,delta:0.1,hits:10,cascade:["c1/cascade"]},
								faces:	 {scale:0.05,dim:100,delta:0.1,hits:10,cascade:["haarcascade_frontalface_alt","haarcascade_eye_tree_eyeglasses"]}
						}};

						itau[0].job = "test.jpg";
						console.log(parm);

						for (var n=0,N=1;n<N;n++)  // program N>1 to test reprogram
							console.log(`INIT[${n}] = `, ATOMIC.opencv("opencv.Me.Thread1","setup",parm));

						for (var n=0,N=5;n<N;n++) // step N>1 to test multistep
							console.log(`STEP[${n}] = `, ATOMIC.opencv("opencv.Me.Thread1","frame",itau));

						// returns badStep if the cascades were undefined at the program step
						console.log("STEP = ", ATOMIC.opencv("opencv.Me.Thread1","helipads",otau));
						console.log(otau);
						break;

					// python machines fail with "cant find forkpty" if "import cv2" attempted

					case "py1": // program python machine
						parm =	{ 
							tau:	[{job:"redefine on run"}],
							ports: {	
						}};
						pgm = `
							print 'Look mom - Im running python!'
							print tau
							tau = [{'x':[11,12],'y':[21,22]}]
							`;

						// By default python attempts to connect to mysql.  
						// So, if mysql service not running or mysql.connector module not found, this will not run.
						console.log({py:pgm, ctx: parm});
						console.log("INIT = ", ATOMIC.python("py1.thread",pgm,parm));
						console.log(parm.tau);
						break;

					case "py2": // program and step python machine 
						parm =	{ 
							tau:	[{job:"redefine on run"}],
							ports: { 	
								frame:	 {},
								helipads:{scale:1.01,dim:100,delta:0.1,hits:10,cascade:["c1/cascade"]},
								faces:	 {scale:1.01,dim:100,delta:0.1,hits:10,cascade:["haarcascade_frontalface_alt","haarcascade_eye_tree_eyeglasses"]}
						}};

						itau[0].job = "test.jpg";
						pgm = `
							print 'Look mom - Im running python!'
							def frame(tau,parms):
								print parms
								return -101
							def helipads(tau,parms):
								print parms
								return -102
							def faces(tau,parms):
								print parms
								return -103
							`;		
						console.log({py:pgm, ctx: parm});
						console.log("INIT = ", ATOMIC.python("py2.Me.Thread1",pgm,parm));
						// reprogramming ignored
						//console.log("INIT = ", ATOMIC.python("py2.Me.Thread1",pgm,parm));

						for (var n=0,N=1; n<N; n++)
							console.log(`STEP[${n}] = `, ATOMIC.python("py2.Me.Thread1","frame",itau));

						console.log("STEP = ", ATOMIC.python("py2.Me.Thread1","helipads",otau));
						break;

					case "py3": // program and step python machine string with reinit along the way
						parm =	{ 
							tau:	[{job:"redefine on run"}],
							ports: {	
								frame:	 {},
								helipads:{scale:1.01,dim:100,delta:0.1,hits:10,cascade:["c1/cascade"]},
								faces:	 {scale:1.01,dim:100,delta:0.1,hits:10,cascade:["haarcascade_frontalface_alt","haarcascade_eye_tree_eyeglasses"]}
						}};

						itau[0].job = "test.jpg";
						pgm = `
							print 'Look mom - Im running python!'
							def frame(tau,parms):
								print parms
								return -101
							def helipads(tau,parms):
								print parms
								return -102
							def faces(tau,parms):
								print parms
								return -103
							`;

						console.log({py:pgm, ctx: parm});
						console.log("INIT = ", ATOMIC.python("py3",pgm,parm));
						console.log("STEP = ", ATOMIC.python("py3","frame",itau));
						// reprogramming ignored
						//console.log("REINIT = ", ATOMIC.python("py3",pgm,parm));
						//console.log("STEP = ", ATOMIC.python("py3","frame",itau));
						console.log(otau);
						break;

					case "js": // program and step a js machine string
						parm =	{ 
							ports: {	
								frame:	 {},
								helipads:{scale:1.01,dim:100,delta:0.1,hits:10,cascade:["c1/cascade"]},
								faces:	 {scale:1.01,dim:100,delta:0.1,hits:10,cascade:["haarcascade_frontalface_alt","haarcascade_eye_tree_eyeglasses"]}
						}};

						itau[0].job = "test.jpg";
						pgm = `
							CON.log('Look mom - Im running javascript!');
							function frame(tau,parms) { 
								CON.log("here I come to save the day");
								tau[0].xyz=123; 
								return 0; 
							}
							function helipads(tau,parms) { 
								tau[0].results=666; 
								return 101; 
							}
							function faces(tau,parms) { return 102; }
							`;

						console.log({py:pgm, ctx: parm});
						console.log("INIT = ", ATOMIC.js("mytest",pgm,parm));
						// frame should return a 0 = null noerror
						console.log("STEP = ", ATOMIC.js("mytest","frame",itau));
						console.log(itau);
						// helipads should return a 101 = badload error
						console.log("STEP = ", ATOMIC.js("mytest","helipads",otau));
						console.log(otau);
						break;	
				}

				res( "thanks!" );
			}
		},

		mysql: {
			host: ENV.MYSQL_HOST,
			user: ENV.MYSQL_USER,
			pass: ENV.MYSQL_PASS
		}

	}, function (err) {
		Trace( "Unit test my engines with /test?config=cv | py1 | py2 | py3 | js" );
	});

	var ATOMIC = require("../atomic").config({
		thread: TOTEM.thread
	});

ATOMIC.paths

Paths to various things.

Kind: static property of ATOMIC
Cfg: Object

ATOMIC.macs

Number of worker cores (aka threads) to provide in the cluster. 0 cores provides only the master.

Kind: static property of ATOMIC

ATOMIC.db

Next available core

Kind: static property of ATOMIC
Cfg: Number

ATOMIC.errors

Error messages

Kind: static property of ATOMIC
Cfg: Object

ATOMIC.config()

Configure the engine interface and estblish workers.

Kind: static method of ATOMIC
Cfg: function

ATOMIC.run()

Run an engine.

	Allocate the supplied callback cb(core) with the engine core that is/was allocated to a Client.Engine.Type.Instance
	thread as defined by this request (in the req.body and req.log).  If a workflow Instance is
	provided, then the engine is assumed to be in a workflow (thus the returned core will remain
	on the same compile-step thread); otherwise, the engine is assumed to be standalone (thus forcing
	the engine to re-compile each time it is stepped).

	As used here (and elsewhere) the terms "process", "engine core", "safety core", and "worker" are 
	equivalent, and should not be confused with a physical "cpu core".  Because heavyweight 
	(spawned) workers run in their own V8 instance, these workers can tollerate all faults (even 
	core-dump exceptions). The lightweight (cluster) workers used here, however, share the same V8 
	instance.  Heavyweight workers thus provide greater safety for bound executables (like opencv and 
	python) at the expense of greater cpu overhead.  

	The goal of hyperthreading is to balance threads across cpu cores.  The workerless (master only)
	configuration will intrinsically utilize only one of its underlying cpu cores (the OS remains, 
	however, free to bounce between cpu cores via SMP).  A worker cluster, however, tends to 
	balance threads across all cpu cores, especially when the number of allocated workers exceeds
	the number of physical cpu cores.

	Only the cluster master can see its workers; thus workers can not send work to other workers, only
	the master can send work to workers.   

	This method will callback cb(core) with the requested engine core; null if the core could not
	be located or allocated.

Kind: static method of ATOMIC

ATOMIC.save()

Save context tau tokens into job files.

Kind: static method of ATOMIC

ATOMIC.insert(req, res)

Provides engine CRUD interface: step/insert/POST, compile/update/PUT, run/select/GET, and free/delete/DELETE.

Kind: static method of ATOMIC

| Param | Type | Description | | --- | --- | --- | | req | Object | Totem request | | res | function | Totem response |

ATOMIC.delete(req, res)

Provides engine CRUD interface: step/insert/POST, compile/update/PUT, run/select/GET, and free/delete/DELETE.

Kind: static method of ATOMIC

| Param | Type | Description | | --- | --- | --- | | req | Object | Totem request | | res | function | Totem response |

ATOMIC.select(req, res)

Provides engine CRUD interface: step/insert/POST, compile/update/PUT, run/select/GET, and free/delete/DELETE.

Kind: static method of ATOMIC

| Param | Type | Description | | --- | --- | --- | | req | Object | Totem request | | res | function | Totem response |

ATOMIC.update(req, res)

Provides engine CRUD interface: step/insert/POST, compile/update/PUT, run/select/GET, and free/delete/DELETE.

Kind: static method of ATOMIC

| Param | Type | Description | | --- | --- | --- | | req | Object | Totem request | | res | function | Totem response |

ATOMIC.mixContext()

Callback engine cb(ctx) with its state ctx primed with state from its ctx.Entry, then export its ctx state specified by its ctx.Exit. The ctx.sqls = {var:"query...", ...} || "query..." enumerates the engine's ctx.Entry (to import state into its ctx before the engine is run), and enumerates the engine's ctx.Exit (to export state from its ctx after the engine is run). If an sqls entry/exit exists, this will cause the ctx.req = [var, ...] list to be built to synchronously import/export the state into/from the engine's context.

Kind: static method of ATOMIC

ATOMIC~ATOMIC

Kind: inner property of ATOMIC
Cfg: Object

Contacting, Contributing, Following

Feel free to


© 2012 ACMESDS