Server WCF Timing Data in AWS Elasticsearch

Project Goals

• Use open source tools and resources to visualize P1 service timing data
• Get better insight into actual customer system load patterns, to help optimize performance and stability

Open Source Tools and Technologies

• Latest AWS Elasticsearch and Kibana service version 5.1
• NodeJS and various NPM modules (7-Zip, D3, and elastic-import)
• Powershell
• Visual Studio Code
• Git, GitHub https://github.com/novac182/super-journey, and npmjs.com

npm install super-journey

How we extracted the data from Customer Support share

• Powershell script using 7-Zip command line options
• Extracted 10 customers' WCF CADS timing logs, for 10 days
• (Not loading data into S3)

7za.exe e "\\P1Customer_logs\XYZ\1-ServerLogs\2017\04\*.zip" -r *CADS_ServerWCFCallSummary.csv -p****** -aou -o*

How we processed the data

• Wrote NodeJS program read the CSV files
• Used D3 module to summarize the statistics by hour
• Output the result into CSV file

  node parseWcf.js <fileName> <outFileName> [mode] [serverTag] [customerTag] [processTag]

  // Rollup WCF timing statistics by hour 
  d3.nest()
    .key(function(d) { return d.DateHour + ',' + d.Service + ',' + d.Operation; })
    .rollup(function(v) { 
      return {
        count: v.length,
        sum: Math.round(d3.sum(v, function(d) { return d.Timing; })),
        avg: Math.round(d3.mean(v, function(d) { return d.Timing; })),
        max: Math.round(d3.max(v, function(d) { return d.Timing; }))
      } 
    })
    .entries(data);

How we loaded the data into AWS ES

• Used elastic-import Node module as ES REST API client
• Loaded each CSV summary file into a new ES index
• (Not using Logstash)

elastic-import.cmd input.csv abc.us-east-1.es.amazonaws.com servertimings wcfcallsummary --csv -h -p

How we visualize the data

• Used Kibana to build various charts and dashboards
• Find patterns and outliers in the data

Impacts

• Can help devlopers identify services which can be optimized
• Gives insight into customer usage behavior
• Next steps - Add other types of services (CS, GIS, etc.)