Graphite Feeder

Introduction

Feed metrics from several sources to graphite.

Installation

No node package deployed to npm yet. Please see section about development to build app yourself.

Configuration

node-config is used for managing configurations.

Configurations are stored in configuration files within your application, and can be overridden and extended by environment variables, command line parameters, or external sources.

Development

Setup docker container running graphite docker-graphite-statsd

sudo docker run -d \
  --name graphite \
  -p 80:80 \
  -p 2003:2003 \
  -p 8125:8125/udp \
  hopsoft/graphite-statsd

If you are developing on MacOS you need to have boot2docker installed and running to be able to use docker.

To access the graphite web UI go to http://localhost/

On MacOS you need to use the proper IP of the boot2docker container (boot2docker ip) instead.

Clone repo and change to directory

git clone [email protected]:commercetools/graphite-feeder.git graphite-feeder && cd $_

Create a configuration file config/local.yaml and adjust missing values.

# configuration for graphite/carbon
graphite:
  host: localhost # use boot2docker IP on MacOS
  port: 2003 # carbon port

# configuration for Salesforce feeder
salesforce:
  username: # required when feeding from Salesforce
  password: # required when feeding from Salesforce
  token: # required when feeding from Salesforce

Resolve and install required dependencies.

npm install

Build and run salesforce feeder (using DEBUG level for logging)

grunt && DEBUG=* ./bin/feeder-salesforce -d

Release npm package using the predefined grunt task.

grunt release

Docker

There is also a docker container for easy setup/execution of the CLI.

Run docker container:

docker run -v /path/to/config/:/config -e NODE_CONFIG_DIR=/config/ sphereio/graphite-feeder

Set an alias for repeated calls:

alias feeder='docker run -v /path/to/config/:/config -e NODE_CONFIG_DIR=/config/ sphereio/graphite-feeder'

After setting the alias you can simply use:

feeder

Feeders

Feeders are used for quering data from different sources and feeding the data into graphite. The following section describes the available feeders.

Salesforce

The Salesforce feeder queries data using Salesforce API and feeds the metric to graphite.

Sample configuration to push a metric to graphite. The configuration defines for each data you want to query a SOQL (Salesforce Object Query Language) as well as the mapping to resulting graphite metric.

# configuration for Salesforce feeder
salesforce:
  prefix: salesforce # prefix for graphite path
  datasources: # sources
    revenue: # query invoiced opportunities
      query: SELECT Account.name, Amount, CurrencyIsoCode,Type FROM Opportunity WHERE FiscalYear=2015 AND (StageName='Invoice' OR StageName='Receipt of payment')
      mapping: # configure mapping salesforce result -> graphite metric
        path: de.commercetools.opportunities.#{Account.Name}.#{Type}.amount.#{CurrencyIsoCode} # define path (field names from result allowed)
        value: Amount # value field in result
        timestamp: CloseDate # date field in result

You can define multiple data sources if you like to query and push different metrics from Salesforce to graphite.

# configuration for Salesforce feeder
salesforce:
  prefix: salesforce # prefix for graphite path
  datasources: #
    <key>: ...
    <key>: ...