@razee/watchkeeper
v1.0.4
Published
Observe Kubernetes changes
Downloads
27
Keywords
Readme
Watch-Keeper
Watch-Keeper is a tool that inventories and reports back the resources running on your cluster. Watch-Keeper works with RazeeDash to display information about your resources.
Install
Via RazeeDash
- Install RazeeDash or use a hosted razee such as razee.io.
- Add your Github org to your razee.
- Go to
https://<razeedash-url>/<your-org-name>/org
then copy and run thekubectl command
against your new cluster to install the watch-keeper components.
Manually
kubectl create cm watch-keeper-config --from-literal=RAZEEDASH_URL=<path-to-razeedash-api> --from-literal=START_DELAY_MAX=0
- eg.
kubectl create cm watch-keeper-config --from- literal=RAZEEDASH_URL=http://app.razee.io/api/v2 --from-literal=START_DELAY_MAX=0
- eg.
kubectl create secret generic watch-keeper-secret --from-literal=RAZEEDASH_ORG_KEY=<plain-text-org-api-key-to-auth-with-razeedash>
- eg.
kubectl create secret generic watch-keeper-secret --from-literal=RAZEEDASH_ORG_KEY=orgApiKey-88888888-4444-4444-4444-121212121212
- eg.
kubectl apply -f https://github.com/razee-io/Watch-keeper/releases/latest/download/resource.yaml
Collecting Resources
Collection Methods
- Watches: this is where watch-keeper gets its name. Watch-keeper creates watches
on any resource with the label
razee/watch-resource=<level>
, and reports to razeedash whenever a change occurs. - Polling: any resource with the
razee/watch-resource=<level>
label is reported. This is useful for resources that are not watchable. - Namespaces: you can gather info from a cluster by labeling a namespace with
razee/watch-resource=<level>
. This will collect and report all data within the labeled namespace at the desired<level>
. Info is only gathered on the polling cycle. See include/exclude lists to limit what is collected. - Non-Namespaced Resources: you can gather info about resources that are not bound
to a namespace by adding the key
poll
to thewatch-keeper-non-namespaced
ConfigMap. Info is only gathered on the polling cycle. See include/exclude lists to limit what is collected. See Non-Namespaced Resources for more info. - Watch by Resource: this allows you to watch and see immediate updates on any resource kind. This can be useful to watch for changes on non-namespaced resource, such as nodes or namespaces, without having to label each resource individually. This can also be useful to watch a single resource type, such as deployments, across the whole cluster. See Watch By Resource for more info.
- Ex.
kubectl label cm my-cm razee/watch-resource=lite
Collection Levels
lite
: reports the resource.metadata
and.status
(where applicable) sections to RazeeDash.detail
: reports the entire resource to RazeeDash, but redacts all environment variables from resources and data values from ConfigMaps and Secrets.debug
: reports the entire resource to RazeeDash. All data is reported, including Secret values.
Notes
<levels>
must be lower case- Labeling namespaces, especially using the detail or debug level collections, can gather much more data than anticipated resulting in delays in data reporting.
- Similarly, delays can occur when reporting on a namespace with lots of resources (> thousand).
Watch By Resource
In order to avoid having to label each individual resource, we allow watching by resource kind. Note: include/exclude lists do not affect watching.
To watch a resource kind, add it to the watch-keeper-non-namespaced
ConfigMap
in the form apiVersion_kind
(where any /
is replaced with an _
), with the
value being the collection levels at which you want to watch
at.
Note: When modifying the watch-keeper-non-namespaced
ConfigMap after the pod
has started, it will take up to VALIDATE_INTERVAL
minutes (default 10m) to start
watching a resource newly added to the ConfigMap, and up to CLEAN_START_INTERVAL
minutes (default 1440m) when changing the collection levels (ie. lite
to detail
).
To get watch-keeper to pick up the changes immediately, kill the watch-keeper
pod to force a restart.
apiVersion: v1
kind: ConfigMap
metadata:
name: watch-keeper-non-namespaced
namespace: <watch-keeper ns>
data:
# 'poll' is optional. Use when you want to poll all non-namespaced resources.
poll: lite
# resources to watch
v1_node: detail
v1_namespace: lite
apps_v1_deployment: lite
v1_configmap: detail
# ... etc.
Non-Namespaced Resources
In order to avoid having to label each individual non-namespaced resource (eg. nodes,
namespaces, customresourcedefinitions), we allow polling of all non-namespaced resources.
This mechanism is similar to how our namespace resource collection works, where
you can label a namespace and we collect all the resources within that namespace
for you; you can think of this like you are labeling the non-namespaced-resources
namespace.
Also similar to how you can label a namespace, there may be resources that you do not want to collect (eg. storageclass), so you should use include/exclude lists to limit what is collected. Note: using the include/exclude list will affect all resources polled, namespaced and non-namespace.
apiVersion: v1
kind: ConfigMap
metadata:
name: watch-keeper-non-namespaced
namespace: <watch-keeper ns>
data:
poll: lite
Include/Exclude Lists
You can include or exclude resources by modifying the ConfigMap named
watch-keeper-limit-poll
, in the namespace your Watch-Keeper is running.
- If both an
include
andexclude
key are specified, onlyinclude
will be used. - The include/exclude list is employed during the Polling, Namespace and Non-Namespaced collection methods. Any individual resource specifically labeled to be watched will still be watched, regardless of the include/exclude list.
Creating a Include/Exclude List
- To create your include/exclude list, the ConfigMap will specify the kind of list you want as the first key, and the rest of the ConfigMap entries become the include/exclude list.
- The include/exclude list itself is created from the ConfigMap keys:
- The keys will be
apiVersion_kind
(where any/
is replaced with an_
). - The value must be
'true'
.
- The keys will be
apiVersion: v1
kind: ConfigMap
metadata:
name: watch-keeper-limit-poll
namespace: <watch-keeper ns>
data:
# Type of list (must be 'true')
include: 'true'
# Resources affected (must be 'true')
v1_node: 'true'
v1_namespace: 'true'
apps_v1_deployment: 'true'
v1_configmap: 'true'
# ... etc.
Feature Intervals
Watch-Keeper collects and reports on data in a few different ways. Each of these
ways is on a differently timed interval and affects when data populates/updates
in RazeeDash. These intervals are configurable via environment variables defined
in the deployment yaml (note: Intervals are in minutes and should follow:
CLEAN_START_INTERVAL > POLL_INTERVAL > VALIDATE_INTERVAL
. ie. 1440 > 60 > 10).
- Heartbeat: every heartbeat collects the user defined cluster metadata,
the cluster id, and the cluster kube version, and sends the data to RazeeDash.
- Timing:
1 minute
(non configurable)
- Timing:
- Validate Watched Resources: every
VALIDATE_INTERVAL
minutes, watch keeper will make sure it has a watch created for the resource kinds (eg. apps/v1 Deployment) that have at least one resource instance with the label. This means the first time you add the label to a previously unwatched resource kind, it could take up toVALIDATE_INTERVAL
minutes to show in razeedash.- Timing:
VALIDATE_INTERVAL=10
- Timing:
- Poll labeled Resources: every
POLL_INTERVAL
minutes, watch keeper will find all resources with therazee/watch-resource
label and send to RazeeDash, as well as find all namespaces with therazee/watch-resource
label and collects/reports all resources within those namespaces.- Timing:
POLL_INTERVAL=60
- coming soon: RazeeDash will have a way to force a re-polling. This will be communicated during the heartbeat, and may take a minute to occur.
- Timing:
- Clean Start: this is a housekeeping interval. It clears out all watches, and
re-verifies watches it should have. Default is once a day.
- Timing:
CLEAN_START_INTERVAL=1440
- Timing:
Cluster Metadata
You can add extra cluster metadata to send to RazeeDash. This can help differentiate
clusters on RazeeDash and be more human readable than a uuid. To do this, add the
label razee/cluster-metadata=true
to a configmap. If the configmap contains the
key name
, RazeeDash will display the name instead of the uuid.
kubectl create cm my-watch-keeper-cm --from-literal=name=mySpecialDevCluster
kubectl label cm my-watch-keeper-cm razee/cluster-metadata=true
Resource Metadata
You can add extra annotations to your resources in order to help the RazeeDash dashboard link to your change management system.
- Working with github:
kubectl annotate <resource-kind> <resource-name> "razee.io/git-repo=<github-repo>"
- ie.
"razee.io/git-repo=https://github.com/razee-io/Watch-keeper"
- ie.
kubectl annotate <resource-kind> <resource-name> "razee.io/commit-sha=<github-sha>"
- ie.
"razee.io/commit-sha=c6645609f8d3b8a48d53246fb7c1f6b60d054aef"
- ie.
- Note: We find it best practice to collect this info and add them to your resource yamls at build time instead of doing it manually on your cluster.
- Working with any change management system:
kubectl annotate <resource-kind> <resource-name> "razee.io/source-url=<fully-qualified-path>"
- ie.
"razee.io/source-url=https://github.com/razee-io/Watch-keeper/commit/c6645609f8d3b8a48d53246fb7c1f6b60d054aef"
- ie.
- Note: We find it best practice to collect this info and add them to your resource yamls at build time instead of doing it manually on your cluster.