Dashflow

Automate local development tasks. See It In Action

What can Dashflow do for you

• Run many commands in background and collect their stdout/stderr
• Automatically trigger new commands when output from those commands matches certain pattern
• Serve a web dashboard page to visualize the status of those commands
• Replace Makefile using a easy readable dashflow.yml YAML file

Installation

Make sure you have installed NodeJS.

# install the package
npm install -g dashflow

Usage

# print help
$ dashflow --help

# by default read from current folder dashflow.yml
$ dashflow

# support multi config files, config will be merged
$ dashflow -c service1/dashflow.yml -c service2/dashflow.yml

# custom http port
$ dashflow -p 9528

# use dashflow shell
dashflow-shell~$ help

  Commands:

    help [command...]          Provides help for a given command.
    exit                       Exits application.
    events [pattern]           Dump events
    emit <event>               Emit a new event
    tail <streamOrWorkflowId>  Tail to a stream or workflow
    attach <streamID>          Attach to a stream
    restart <streamID>         Restart a stream
    env                        Print environment variables

Concepts

Dashflow is modeled around some key concepts.

Please take some time to get familiar with those concepts so you can use the tool more effectively.

Event

An event is just a plain string, associated with its creation time. Usually events will have certain prefixes so that they can easily be distinguished from different sources of events.

For example, here are some sample events:

stream:watch-src:watch:add:src/new_file
stream:watch-test:watch:addDir:src/test/
command:lint:shell:yarn --silent lint:stdout:lint passed
workflow:initial-lint:command:lint:shell:yarn --silent lint:stdout:lint passed

In dashflow, all events store in memory (which means all events will be lost once the dashflow process exited).

The default view of events are ordered by their creation time, newer to older.

Command

A command is an alias for a shell function, commands can be composed together, executing one by one or concurrently.

The following is an example of command configurations:

commands:
  // the following concise way of declaring a command is a shortcut for the formal one
  // which is demonstrated with "test" command
  lint: yarn --silent lint
  test:
    shell: { cmd: yarn --silent test --no-color --verbose }
  lint-then-test:
    // run those commands one by one
    serial:
      - lint
      - test
  lint-and-test:
    // run those commands concurrently
    parallel:
      - lint
      - test
  list-web-files:
    // by default we assume shell should executes from the same folder
    // where dashflow.yml file is located
    // but we can specify a sub folder by providing 'cwd' argument to shell
    shell: { cmd: ls, cwd: web }

Stream

A stream is something that will emit events, it can be either a long running process like a web server, or a one off command like running test.

The following is an example of stream configurations:

streams:
  watch-lib:
    // listen on current folder for file changes and report evens on matching files
    watch: { glob: "lib/**/*.js" }
  watch-test:
    // can specify a sub folder by providing 'cwd' argument to watch
    watch: { glob: "**/*.js", cwd: test }
  python:
    // can execute a shell command
    shell: { cmd: python -i }
  vim:
    // can specify a sub folder by providing 'cwd' argument to shell
    shell: { cmd: vim, cwd: test }

Workflow

A workflow represents a rule, typically some execution logic like "when A happens, then B then C then D".

A concrete example is that when a file changed in "src" folder, we would like lint and test to be triggered automatically.

A workflow also emit events, from this point of view it is also a special stream.

In dashflow, workflows are always triggered while a new event matches given pattern.

The following is an example of workflow configurations:

workflows:
  initial-lint:
    // special event fires when dashflow starts
    match: SYSTEM:started
    command: lint
  initial-test:
    match: SYSTEM:started
    command: test
  lib-lint-then-test:
    match: watch-lib:.*
    // execute those commands serially
    serial:
      - command: lint
      - command: test
  test-lint-then-test:
    match: watch-test:.*
    // execute those commands parallelly
    parallel:
      - command: lint
      - command: test
      - restart: irb
  web-lint:
    match: watch-web:.*
    serial:
      - delay: 1000
      - command: lint

Dashboard

Dashflow starts an HTTP server when running in daemon mode, and what being served is an special page that visualizes all those events dashflow has collected. The page has many tabs, each tab is a dashboard. A dashboard has many widgets, each occupies a rectangular area.

Different widget type behaves very differently:

• a "log" widget shows a subset of all events by applying a filter
• a "gauge" widget shows status texts base on calculations on event streams, like "passing/failed/unknown" etc
• a "banner" widget displays static text

For example:

dashboards:
  all-in-one:
    - log:
        position: 0 0 45% 50%
        title: Lint
        filter: command:lint:.*
    - log:
        position: 45% 0 90% 50%
        title: Test
        filter: command:test:.*
    - banner:
        position: 90% 0 100% 16%
        content: This is Dashflow Dashboard
    - gauge:
        position: 90% 16% 100% 32%
        title: Lint Status
        filter: command:lint:state:.*
        scan:
          when:
            - pattern: started
              text: Running
              level: warning
            - pattern: exited with 0
              text: Passed
              level: success
            - pattern: exited with
              text: Failed
              level: error
          default:
            text: Unknown

Configuration

A dashflow.yml is required in order to utilize dashflow as a local development workflow orchestrator.

Basically what we need to do is to model the workflow we already have into those concepts in dashflow.

Here're some example configurations for your inspiration.

dashflow.yml for a frontend project

commands:
  lint: yarn --silent lint_min
  test: yarn --silent test_min

streams:
  site: { shell: { cmd: yarn --silent site } }
  watch-src: { watch: { glob: "src/**/*.js*" } }

workflows:
  initial-lint:
    match: SYSTEM:started
    command: lint
  initial-test:
    match: SYSTEM:started
    command: test
  on-src-update: |
    match: watch-src:.*
    parallel:
      - command: lint
      - command: test

dashboards:
  spark-ui:
    - log:
        position: 0 0 90% 50%
        title: Webpack Logs
        filter: stream:site:.*
    - gauge:
        position: 90% 0 100% 25%
        title: Lint Status
        filter: command:lint:state:.*
        scan:
          when:
            - pattern: started
              text: Running
              level: warning
            - pattern: exited with 0
              text: Passed
              level: success
            - pattern: exited with
              text: Failed
              level: error
          default:
            text: Unknown
    - gauge:
        position: 90% 25% 100% 50%
        title: Test Status
        filter: command:test:state:.*
        scan:
          when:
            - pattern: started
              text: Running
              level: warning
            - pattern: exited with 0
              text: Passed
              level: success
            - pattern: exited with
              text: Failed
              level: error
          default:
            text: Unknown
    - log:
        position: quadrant/bottom-left
        title: Lint
        filter: command:lint:.*
    - log:
        position: quadrant/bottom-right
        title: Test
        filter: command:test:.*

dashflow.yml for dashflow project itself

Click here

Reference

commandID/streamID/workflowID is the key of a command/stream/workflow definition in the configuration file.

Command

# execute shell command
commandID:
  shell:
    cmd: <shell command>
    cwd: "working folder, default to be where dashflow.yml is located"
# produces events in following formats
#   command:commandID:shell:<shell command>:stdout:<shell stdout>
#   command:commandID:shell:<shell command>:state:started
#   command:commandID:shell:<shell command>:state:exited with <shell exit code>

Stream

# listen on file changes
streamID:
  watch: 
    glob: <glob pattern>
    cwd: "working folder, default to be where dashflow.yml is located"
    ignore: /regex pattern for files to ignore/
# produces events in following formats
#   stream:streamID:watch:add:<file path>
#   stream:streamID:watch:addDir:<file path>
#   stream:streamID:watch:change:<file path>
#   stream:streamID:watch:unlink:<file path>
#   stream:streamID:watch:unlinkDir:<file path>

# execute shell command
streamID:
  shell:
    cmd: <shell command>
    cwd: "working folder, default to be where dashflow.yml is located"
# short form for the above if cwd is default
streamID: <shell command>
# produces events in following formats
#   stream:streamID:shell:<shell command>:stdout:<shell stdout>
#   stream:streamID:shell:<shell command>:state:started
#   stream:streamID:shell:<shell command>:state:exited with <shell exit code>

Workflow

# execute command if there's a match
workflowID:
  match: "event pattern that triggers this workflow"
  command: "command name to trigger"
# produces events in following formats
#   workflow:workflowID:<command event format here>

# restart stream if there's a match
workflowID:
  match: "event pattern that triggers this workflow"
  restart: <streamID>
# produces events in following formats
#   workflow:workflowID:restart:<streamID>

# wait for a certain time if there's a match
# usually used together with serial command
workflowID:
  match: "event pattern that triggers this workflow"
  wait: <time in milliseconds>
# produces no events

# run workflow actions serially
workflowID:
  match: "event pattern that triggers this workflow"
  serial: 
    - command: <cmd1>
    - wait: 1000
    - command: <cmd2>

# run workflow actions parallelly
workflowID:
  match: "event pattern that triggers this workflow"
  parallel: 
    - command: <cmd1>
    - command: <cmd2>

Dashboard

# show a subset of all event streams by applying a filter
dashboardID:
  - log:
      position: <rectangular: x1 y1 x2 y2> OR <position-alias>
      title: <title string>
      filter: <filter pattern>

# show a console log with a header gauge
dashboardID:
  - log:
      position: <rectangular: x1 y1 x2 y2> OR <position-alias>
      title: <title string>
      filter: <log filter pattern>
      gauge:
        filter: <gauge filter pattern>
        scan:
          when:
            - pattern: started
              text: Running
              level: warning
            - pattern: exited with 0
              text: Passed
              level: success
            - pattern: exited with
              text: Failed
              level: error
          default:
            text: Unknown

# show status text by running some calculations
dashboardID:
  - gauge:
      position: <rectangular: x1 y1 x2 y2> OR <position-alias>
      title: <title string>
      filter: <filter pattern>
      scan:
        when:
          - pattern: <pattern 1>
            text: <status text 1>
            level: <status level: success|warning|error>
          - pattern: <pattern 2>
            text: <status text 2>
            level: <status level: success|warning|error>
        default:
          text: <default text>
          level: <default level, default: none>

# display static content
dashboardID:
  - banner:
      position: <rectangular: x1 y1 x2 y2> OR <position-alias>
      content: <static content string>

Reference

Position Aliases

+----------------------------------------+
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
+----------------------------------------+

              fullscreen

+----------------------+-----------------------+
|                      |                       |
|                      |                       |
|                      |                       |
| quadrant/top-left    |   quadrant/top-right  |
|        OR            |          OR           |
|     quadrant/2       |       quadrant/1      |
|                      |                       |
+----------------------------------------------+
|                      |                       |
|                      |                       |
|                      |                       |
| quadrant/bottom-left | quadrant/bottom-right |
|        OR            |        OR             |
|    quadrant/3        |      quadrant/4       |
|                      |                       |
|                      |                       |
+----------------------+-----------------------+

                  quadrant

+----------------------------------------+
|                                        |
|                                        |
|                                        |
|              quadrant/top              |
|                                        |
|                                        |
|                                        |
+----------------------------------------+
|                                        |
|                                        |
|                                        |
|                                        |
|             quadrant/bottom            |
|                                        |
|                                        |
|                                        |
+----------------------------------------+

                  quadrant

+-------------------+--------------------+
|                   |                    |
|                   |                    |
|                   |                    |
|                   |                    |
|                   |                    |
|                   |                    |
|    quadrant/left  |    quadrant/right  |
|                   |                    |
|                   |                    |
|                   |                    |
|                   |                    |
|                   |                    |
|                   |                    |
|                   |                    |
+-------------------+--------------------+

                  quadrant

Contributors

Your contributions are highly welcomed! Please check out Contributor's Guide for more details.

Acknowledgement

Dashflow was initially built by engineers @ FreeWheel, a Comcast company, and donated to the open source community.