Sematext Docker Agent

Sematext Docker Agent collects Metrics, Events and Logs from the Docker API for SPM Docker Monitoring & Logsene / Hosted ELK Log Management. Works with CoreOS, RancherOS, Docker Swarm, Kubernetes, Apache Mesos, Hashicorp Nomad, Amzon ECS, ... see installation.

How it works

Video: Log Management for Docker

Gathered information:

• Operating System Metrics of the Host machine (CPU / Mem / Swap/ ...)
• Docker Container Metrics/Stats
  • CPU Usage
  • Memory Usage
  • Network Stats
  • Disk I/O Stats
• Docker Events
  • Version Information on Startup:
    • server-info – created by spm-agent framework with node.js and OS version info on startup
    • docker-info – Docker Version, API Version, Kernel Version on startup
  • Docker Events:
    • Container Lifecycle Events like
      • create, exec_create, destroy, export
    • Container Runtime Events like
      • die, exec_start, kill, pause, restart, start, stop, unpause, ...
• Docker Logs
  • default fields
    • hostname / IP address
    • container id
    • container name
    • image name
    • message
  • Log format detection and log parsers: - NGINX - Apache httpd, Kafka, Solr, HBase, Zookeeper, Cassandra - MySQL - MongoDB - Redis - Elasticsearch - Nsq.io - JSON, ...

Installation

1. Get a free account at sematext.com/spm

2. Create an SPM App of type "Docker" and copy the SPM Application Token

   • For logs (optional) create a Logsene App to an App Token for Logsene

3. Run the image

   docker pull sematext/sematext-agent-docker
   docker run -d --name sematext-agent-docker -e SPM_TOKEN=YOUR_SPM_TOKEN -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN  -e HOSTNAME  -v /var/run/docker.sock:/var/run/docker.sock sematext/sematext-agent-docker
   # Alternative TCP: default TCP 2375 on 'localhost' (=> container gateway address), -v is not required
   docker run -d --name sematext-agent-docker -e SPM_TOKEN=YOUR_SPM_TOKEN -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN  -e HOSTNAME  sematext/sematext-agent-docker

   Required Parameters:

   • -e SPM_TOKEN - SPM Application Token
   • -e HOSTNAME - name of the docker host e.g. '$HOSTNAME' for Amazon ECS see HOSTNAME_LOOKUP_URL
   • -v /var/run/docker.sock - path to the docker socket (optional, if dockerd provides TCP on 2375, see also DOCKER_PORT and DOCKER_HOST paramter)

   TCP and TLS connection (> image version 1.29.13)

   If the unix socket is not available Sematext Agent assumes the Container Gateway Address (autodetect) and port 2375 as default (no TLS) - this needs no configuration. In case the Docker Daemon TCP settings are different, you have to configure the TCP settings. The TCP settings can be modified with the following parameters:

• -e DOCKER_HOST - e.g. tcp://ip-reachable-from-container:2375/ - if not set unix:///var/run/docker.sock or if this does not exists tcp://gateway:2375 will be used. In this case you don't need -v to mount /var/run/docker.sock
• -e DOCKER_PORT - in case Docker TCP connection is used, the agent will use its gateway address (autodetect) with the given DOCKER_PORT
• -e DOCKER_TLS_VERIFY - 0 or 1
• -e DOCKER_CERT_PATH - path to your certificate files, mount the path to the countainer with "-v $DOCKER_CERT_PATH:$DOCKER_CERT_PATH"

Example using docker-machine with Docker Swarm:

docker-machine env --swarm swarm-master
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.101:3376"
export DOCKER_CERT_PATH="/Users/stefan/.docker/machine/machines/swarm-master"
export DOCKER_MACHINE_NAME="swarm-master"
eval "$(docker-machine env swarm-master)"
docker run -d --name sematext-agent --restart=always -e SPM_TOKEN=MY_TOKEN -e HOSTNAME  -e DOCKER_TLS_VERIFY -e DOCKER_CERT_PATH -e DOCKER_HOST -v $DOCKER_CERT_PATH:$DOCKER_CERT_PATH sematext/sematext-agent-docker

**Optional Parameters:**
- --privileged  might be required for Security Enhanced Linux (the better way is to have the right policy ...)
- -e HOSTNAME_LOOKUP_URL - On Amazon ECS, a [metadata query](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) must be used to get the instance hostname (e.g. "169.254.169.254/latest/meta-data/local-hostname")
- -e HTTPS_PROXY - URL for a proxy server (behind firewalls)
- -e LOGSENE_URL - URL for bulk inserts into Logsene. Required only for Logsene On-Premises only.
- -e SPM_URL - URL for bulk inserts into SPM. Required only for SPM On-Premises. 

    **Docker Logs Parameters:**
- -e LOGSENE_TOKEN - Logsene Application Token for logs 
- Whitelist containers for logging
  - -e MATCH_BY_NAME - regular expression to white list container names 
  - -e MATCH_BY_IMAGE - regular expression to white list image names 
- Blacklist containers
  - -e SKIP_BY_NAME - regular expression to black list container names 
  - -e SKIP_BY_IMAGE - regular expression to black list image names for logging 
  - -v /yourpatterns/patterns.yml:/etc/logagent/patterns.yml - to provide custom patterns for log parsing, see [logagent-js](https://github.com/sematext/logagent-js)
- -v /tmp:/logsene-log-buffer a directory to store logs, in case of a network or service outage. Docker Agent deletes this files after successful transmission.  
- -e KUBERNETES set this variable to "1" to parse container names into the fields kubernetes.pod_name, kubernetes.namespace and kubernetes.container_name


You’ll see your Docker metrics in SPM after about a minute.

5. Watch metrics, use anomaly detection for alerts, create e-mail reports and much more ...

Docker Events:

Log Routing

Routing logs from different containers to separate Logsene Apps can be configured via docker labels. Simply tag a container with the label (or environment variable) LOGSENE_TOKEN. Sematext Docker Agent inspects the containers for this Label and ships the logs to the defined Logsene App.

Example: The following command will start nginx webserver and logs for this container will be shipped to the related Logsene App.

docker run --label LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGSENE_TOKEN -p 80:80 nginx
# or use environment variable on Kubernetes (no support for Docker labels)
# docker run -e LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGSENE_TOKEN -p 80:80 nginx

All other container logs will be shipped to the Logsene App specified in the docker run command for sematext/sematext-agent-docker with the environment variable LOGSENE_TOKEN.

Integrated Log Parser

SPM for Docker recognizes log formats - so your logs arrive in a structured format in Logsene! The format recognition, data extractions, date parsing etc. is provided by logagent-js and covers:

• Format detection e.g. for
  • Mongo DB
    • Nginx
    • Apache httpd, Kafka, Cassandra, HBase, Solr, Zookeeper
    • MySQL
    • Redis
• plain text log messages
• line delimited JSON logs
• GeoIP enrichment for webserver logs, or any other field defined in the pattern definitions

To use a custom pattern definition simply mount a volume to '/etc/logagent/patterns.yml':

-v /mydir/patterns.yml:/etc/logagent/patterns.yml

Feel free to contribute to logagent-js to enrich the default pattern set.

Installation on CoreOS Linux

Sematext Agent for Docker can monitor CoreOS clusters including metrics and logs from Docker and journald.
See: Setup Sematext Docker Agent on CoreOS

Installation on Kubernetes

Run Sematext Docker Agent as Kubernetes DaemonSet.

1. Create sematext-agent.yml - and set your SPM and Logsene App Token in the section spec.env.
2. Run the DaemonSet

kubectl create -f sematext-agent.yml --validate=false

Installation on Docker Swarm

Please read Docker Swarm: Collecting Metrics, Events & Logs

Installation on Nomad by Hashicorp

See and example of the job description for Nomad by Hashicorp

Installation on Mesos / Marathon

Please note that you have to specify the number of Mesos nodes (instances), SPM App Token and Logsene App Token. Example call to the Marathon API:

curl -XPOST -H "Content-type: application/json" http://your_marathon_server:8080/v2/apps  -d '
{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "sematext/sematext-agent-docker"
    },
    "volumes": [
      {
        "containerPath": "/var/run/docker.sock",
        "hostPath": "/var/run/docker.sock",
        "mode": "RW"
      }
    ],
    "network": "BRIDGE"
  },
  "env": {
        "LOGSENE_TOKEN": "YOUR_LOGSENE_TOKEN",
        "SPM_TOKEN": "YOUR_SPM_TOKEN" 
  },
  "id": "sematext-agent",
  "instances": 1,
  "cpus": 0.1,
  "mem": 100,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ]
}

Support

1. Please check the SPM for Docker Wiki
2. If you have questions about SPM for Docker, chat with us in the SPM user interface or drop an e-mail to [email protected]
3. Open an issue here
4. Contribution guide here