Shoal

Shoal allows you to group and run multiple processes. Shoal manages these processes and exposes controls and status information via a HTTP JSON API. Aside from the JSON interface there is a Web user interface and a CLI.

Use-cases:

• Centralize process/worker configuration using a single JSON config file (Avoid hundreds of upstart scripts)
• Use a CLI and Web UI to monitor and control a large number of process/worker instances
• Simplifying process/worker management for developers (e.g. local installs of ETL's)
• Smart deployments: Automatically stop workers that have been removed from configuration, and start instances that were added
• Centrally control logging, environment and restart behavior of processes

Status & Security

We provide two authentication mechanisms out of the box (see Authentication below), however these are not configured by default and have not been battle-tested in production environments.

By default, the Shoal Manager (shoald) binds to 127.0.0.1 (loopback device) and allows execution of commands sent to it. This is something to be taken into consideration when running outside of sandboxed environments.

Please be mindful of this when running this in a production environment.

Install

npm install shoal

Usage

In order to deploy and monitor processes with one of the user interfaces, you need to run the Shoal Manager. This is done by executing the shoald command that's included with the NPM module.

By default, the Shoal Manager listens for commands on port 54047 and also binds the Admin UI on port 54047:

$ shoald
Shoal Manager running on http://127.0.0.1:54047
Shoal Admin UI running on http://127.0.0.1:54048

Here are some CLI options that can be given to shoald:

$ shoald --help

  Usage: shoald [options]

  Options:

    -h, --help                     output usage information
    -V, --version                  output the version number
    -c, --config <path>            Run server with this configuration
    -pd, --deploy <path>           Run Manager with this process deployment
    -p, --port <port>              Port to bind on (Default: 54047)
    -b, --bind <host>              Host to bind on (Default: 127.0.0.1)
    -ap, --admin-port <port>       Port to bind Admin UI on (Default: 54048)
    -ab, --admin-bind <host>       Host to bind Admin UI on (Default: 127.0.0.1)
    -ah, --admin-html-file <path>  Path to HTML file to customize Admin UI
    --disable-ui                   Disable Admin UI
    --quiet                        Don't show Shoal info
    --verbose                      More verbose output

Once the Shoal Manager is running process configuration can be deployed with the shoal deploy command:

shoal deploy examples/simple/processes.json

A status of current processes and number of instances can be retrieved using the shoal status command:

shoal status

Alternatively, you can use the Admin Web UI to view and control processes (Defaults to http://localhost:54048/):

Process Configuration

The Shoal Manager takes a JSON configuration that allows us to specify which processes should be run and how they should be executed. In Shoal, a process is a definition of a command that should be daemonized. One process can have many instances that all perform the same job.

Here's an example configuration:

{
  "env": {
    "NODE_ENV": "staging"
  },
  "killTimeoutMs": 30000,
  "logRoot": "/var/log",
  "processes": [
    {
      "name": "Cassandra Worker",
      "cmd": "node",
      "args": ["script/worker.js", "cassandra"],
      "env": {
        "NAME": "$processName",
      },
      "instances": 8,
      "autoRestart": true,
      "autoRestartTimeoutMs": 3000
    },
    {
      "name": "Some Daemon",
      "cmd": "somed",
      "args": ["-f"],
      "killTimeoutMs": 1000,
      "logRoot": "~/log",
      "outFile": "somed.log",
      "errFile": "somed.error",
      "instances": 0
    }
  ]
}

Each process in the processes array takes the following required parameters:

|Parameter |Description | |-------------|-------------------------------------------| |name |Name of the process/worker| |cmd |Unix command to be executed without parameters (can be absolute path too)| |args |Array of arguments to be given to the unix command| |instances |The number of instances of this process that should be run. Can be 0 for inactive process definitions|

The following options can be given either in the top-level JSON or in the indvidual process definitions (overrides global settings):

|Option |Defaults |Description | |-------------------------|---------|--------------------------------------------| |env |{} |Hash with environment variables that should be used. See below for available Macros| |killTimeoutMs | |Number of miliseconds after which a unresponsive process instance (not responding to SIGHUP) should be send a SIGKILL| |autoRestart |false |Automatically restart a process instance after exit| |autoRestartTimeoutMs |1000 |Number of miliseconds to wait until restarting the process instance| |logRoot | |Specify output logging directory. When specified each process will have stdout and stderr send to respective process-name.out and process-name.err files| |outFile |$processShortName.out|Override default stdout output file. Requires logRoot to be set| |errFile |$processShortName.err|Override default stderr output file. Requires logRoot to be set| |quiet |false |Don't show any output from shoal manager server or admin UI|

Environment Macros

When configuring custom environment variables for a process or configuration, you can use the following internal variables/macros:

|Variable |Description | |-----------------|--------------------------------| |$processName |Configured name of the process| |$processShortName|A dashed and lowercase conversion of the processName (e.g. My Worker => 'my-worker')| |$processId |A unique ID of the process based on the name, cmd and env| |$shoalCwd |The working directory path that the Shoal Manager is running in| |$shoalVersion |Shoal version| |$logRoot |The configured log root for the process|

Server/Client Configuration

Using the --config flag you can supply a JSON file with configuration options to the shoald and shoal commands:

|Option |Defaults |Description | |-------------------------|---------|--------------------------------------------| |host |127.0.0.1|Shoal Manager host to bind or connect to| |port |54047|Shoal Manager port to use for binding or connecting| |adminHost |127.0.0.1|Shoal Admin UI host to bind on| |adminPort |54048|Shoal Admin UI port to bind on| |auth ||Authentication options for Shoal Manager (see below)| |adminAuth ||Authentication options for Shoal Admin UI (see below)|

Authentication

Currently we support two methods of authentication: basic and ssl.

Basic Authentication

Here's an example of a server-client configuration with basic authentication:

{
  "auth": {
    "type": "basic",
    "key": "my scecret"
  },
  "disableAdminUi": true
}

Instead of specifying the key in the configuration you can instead use keyFile to point to a file.

Basic auth can also be applied on the Admin UI as follows:

{
  "auth": {
    "type": "basic",
    "key": "my scecret"
  },
  "adminAuth": {
    "type": "basic",
    "username": "admin",
    "key": "secret"
  },
}

SSL Authentication

A more secure way of authenticating is to use SSL. We recommend this authentication strategy.

This requires first setting up the certificates (CA, Server and Client). A convenience script is provided in tools:

bash tools/create-ssl-keys.sh

Once the certificates are made, configure your server:

{
  "auth": {
    "type": "ssl",
    "keyFile": "cert/server.key",
    "certFile": "cert/server.crt",
    "caFile": "cert/ca.crt",
    "passphrase": "shoal"
  },
   "disableAdminUi": true
}

And your client:

{
  "auth": {
    "type": "ssl",
    "keyFile": "cert/client.key",
    "certFile": "cert/client.crt",
    "caFile": "cert/ca.crt",
    "passphrase": "shoal"
  },
  "host": "localhost"
}

A full example of this can be found in examples/ssl-auth.

Future

• Improve default security features
• Allow Shoal to dynamically control the number of instances per process definition based on the 'business' of the running instances

License

(The MIT License)

Copyright (c) 2014 Dominiek ter Heide [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.