kang: distributed system observability

Kang is a facility for debugging networked software services by exposing internal state via a simple HTTP API. Service state is organized into a two-level hierarchy of "objects" organized by "type". For example, a simple HTTP server may have two types of objects: "requests" and "connections", and there may be many objects of each type. Each server defines its own types and the structures of its objects.

Demo

First, install the command-line tool:

# npm install -g kang

Then run the example server in this repo:

# node examples/server.js
server listening at http://0.0.0.0:8080

Now run the kang debugger:

# kang -hlocalhost:8080

Run "help" for some suggested examples and try them out.

kang tool

Usage: kang [-h host1[host2...]]

Remote servers are specified using the following format:

[http[s]://]host[:port][/uri]

All fields other than the host are optional. Nearly any combination may be specified, as in:

  REMOTE HOST              MEANS
  localhost                http://localhost:80/status/snapshot
  localhost:8080           http://localhost:8080/status/snapshot
  localhost:8080/kang      http://localhost:8080/kang
  https://localhost/kang   https://localhost:443/kang

Multiple servers may be specified in a comma-separated list. Servers are specified using the -h option or (if none is present) the KANG_SOURCES environment variable.

When you run kang, it creates a snapshot of the distributed system's state by querying each of the servers. You can browse the state interactively. Type "help" for more information.

Background

While interactive program execution is a useful feature during development, the most important feature for debuggers in both development and production environments is the presentation of current program state. Program state is often examined on an ad-hoc basis by engineers debugging a particular problem, but it's often useful to build tools to automatically analyze this state as well, either to summarize it for humans or to automatically look for certain classes of problems. In this regard, kang is a debugger for distributed systems: it fetches, aggregates, and presents program state for consumption by both humans and automated tools. The goal is to allow each component of the distributed system to describe the objects it knows about (and potentially a small amount of metadata suggesting what to do with this information) so that the kang system can fetch, aggregate, and present this information usefully.

In debugging distributed systems of heterogeneous components, it's critical to be able to quickly understand the internal state of each component. We have https://github.com/trentm/node-bunyan and https://github.com/joyent/node-panic to understand explicit errors and fatal failures, but you need more to understand why a service is simply behaving wrong.

Most of the time, the internal state takes the form of just a few important types of objects. It would be really useful if each service provided a standard way of extracting this state for the purpose of debugging.

API

kang defines a single HTTP entry point, /kang/snapshot, that returns a snapshot of the service's internal state in the form of a JSON object that looks like this:

{
        /* service identification information */
        "service": {
                "name": "ca",
                "component": "configsvc",
                "ident": "us-sw-1.headnode",
                "version": "0.1.0vmaster-20120126-2-g92bf718"
        },

        /* arbitrary service stats */
        "stats": {
                "started": "2012-03-20T17:03:59.221Z",
                "uptime": 86403217,
                "memory": {
                        "rss": 10850304,
                        "heaptotal": 2665280,
                        "heapused": 1700788
                },
                "http": {
                    "nrequests": 1709,
                    "nrequestsbycode": {
                      "200": 1705,
                      "201": 1,
                      "204": 1,
                      "503": 1
                    }
                }
        },

        /* extra service-specific information */
        "types": [ 'instrumentation', 'instrumenter' ],

        "instrumentation": {
                "cust:12345;1": {
		    "creation_time": "2012-01-26t19:20:30.450z",
		    "label": "12345/1"
                        "module": "node",
                        "stat": "httpd_ops",
                        "decomposition": "latency",
                        "granularity": 1,
                        "instrumenters": {
                                "instrumenter:instr1": "enabled",
                                "instrumenter:instr2": "enabled",
                                "instrumenter:instr3": "disabled"
                        }
                }
        },

        "instrumenter": {
                "instr1": {
                        "creation_time": "2012-01-26t19:20:30.450z",
                        "instrumentations": [ "instrumentation:cust:12345;1" ],
                        "last_contact": "2012-01-26t19:20:30.450z"
                },
                "instr2": {
                        "creation_time": "2012-01-26t19:20:30.450z",
                        "instrumentations": [ "instrumentation:cust:12345;1" ],
                        "last_contact": "2012-01-26t19:20:30.450z"
                },
                "instr3": {
                        "creation_time": "2012-01-26t19:20:30.450z",
                        "instrumentations": [ ],
                        "last_contact": "2012-01-10t19:20:30.450z"
                }
        }
}

Note that many of the above field names match the corresponding fields used in Bunyan for logging. Clients can link objects reported by multiple components (or even services) by assuming any given (type, id) tuple is unique. Clients can also link any string of the form "type:id" (for a known object type and id) to the corresponding object. For example, the "instrumenter:instr1" key in the instrumentation above can be linked directly to that object.

In the future we may define semantics for some fields like "label", and "creation_time" so that the tools can present this information more usefully.

Server library

kang includes a server library for implementing the above API. Any project that wants to take advantage need only implement a few entry points:

• report service identification information
• report stats
• list object types
• list objects for a given type
• serialize one object

Services can add information incrementally as desired. The library takes care of formatting this data appropriately.

Client library

kang includes a client library for listing and browsing objects from a set of services. See cmd/kang.js for example usage.

CLI

See above for details.

Future work

• Remove prefixes on library function names