Elastic APM Logger

Setup

Ensure that you have elastic-apm-node installed, so that your NodeJS's app's metrics get sent back to the Elastic Stack. If you haven't then follow this guide first - https://www.elastic.co/guide/en/apm/agent/nodejs/current/express.html

Once this has been done, you can install this library:

npm install elastic-apm-logger

Full Usage Example

Below is an example for how one can use this library.

For ECS formatted logs, please scroll down, there is a separate section which covers that.

Example:

// Constants
const ELASTIC_CLOUD_ID = '*** Elastic Cloud ID Here ***' // This can be found in your https://cloud.elastic.co dashboard
const APM_SERVICE_NAME = 'api-v1' // This is the name of your service.... something meaningful, and has to be a single string
const ELASTICSEARCH_API_KEY = '*** API Key ***' // This is created under Your Deployment > Stack Management > API keys
const APM_SERVER_SECRET_TOKEN = '*** APM Secret Token ***' // This can be found here - https://www.elastic.co/guide/en/apm/guide/current/secret-token.html
const APM_SERVER_URL = ' *** APM Server HTTPS URL ***' // This can be found in your https://cloud.elastic.co dashboard

// Start Elastic APM - METRICS Collection
const apm = require('elastic-apm-node').start({
  serviceNodeName: APM_SERVICE_NODE_NAME,
  serviceName: APM_SERVICE_NAME,
  secretToken: APM_SERVER_SECRET_TOKEN,
  serverUrl: APM_SERVER_URL
})

// Start Elastic APM - LOGGING Collection
const elasticApmLogger = require('elastic-apm-logger')
elasticApmLogger.startLogging({
  esAuthObject: {
    cloud: { id: ELASTIC_CLOUD_ID },
    auth: { apiKey: ELASTICSEARCH_API_KEY }
  },
  serviceName: APM_SERVICE_NAME,
  apmObject: apm
})

You should see your logs coming in now under the Services > Your Service > Logs tab.

If you are looking to format your logs into ECS Format before sending it back, please refer to the next section.

Authentication

Since this library internally uses the @elastic/elasticsearch npm library, we will also then follow the same method of authentication, by accepting an object of the same format.

Please see https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/client-connecting.html#authentication for more details.

It is essentially the object that the Client accepts in the @elastic/elasticsearch npm library.

Here are some examples on how you can authenticate:

Using Cloud ID and API Key:

You can use the Cloud ID and API key to authenticate. This is usually used for Elastic Cloud instances:

// Constants
const ELASTIC_CLOUD_ID = '*** Your Elastic Cloud ID Here ***' // This can be found in your https://cloud.elastic.co dashboard
const APM_SERVICE_NAME = 'api-server-v1' // This is the name of your service.... something meaningful, and has to be a single string
const ELASTICSEARCH_API_KEY = '*** Your API Key ***' // This is created under Your Deployment > Stack Management > API keys
const APM_SERVER_SECRET_TOKEN = '*** Your APM Secret Token ***' // This can be found here - https://www.elastic.co/guide/en/apm/guide/current/secret-token.html
const APM_SERVER_URL = ' *** Your APM Server HTTPS URL ***' // This can be found in your https://cloud.elastic.co dashboard

// Start Elastic APM - METRICS Collection
const apm = require('elastic-apm-node').start({
  serviceNodeName: APM_SERVICE_NODE_NAME,
  serviceName: APM_SERVICE_NAME,
  secretToken: APM_SERVER_SECRET_TOKEN,
  serverUrl: APM_SERVER_URL
})

// Start Elastic APM - LOGGING Collection
const elasticApmLogger = require('elastic-apm-logger')
elasticApmLogger.startLogging({
  esAuthObject: {
    cloud: { id: ELASTIC_CLOUD_ID },
    auth: { apiKey: ELASTICSEARCH_API_KEY }
  },
  serviceName: APM_SERVICE_NAME,
  apmObject: apm
})

Using Username and Password:

You can use username and password to authenticate. This is usually used for self managed Elasticsearch instances (hosted on AWS EC2 etc):

// Constants
const ELASTICSEARCH_DB_URL = '*** Your Elasticsearch database URL Here ***' // This is the url of your elasticsearch database instance
const APM_SERVICE_NAME = 'api-v1' // This is the name of your service.... something meaningful, and has to be a single string
const ELASTICSEARCH_API_KEY = '*** API Key ***' // This is created under Your Deployment > Stack Management > API keys
const APM_SERVER_SECRET_TOKEN = '*** APM Secret Token ***' // This can be found here - https://www.elastic.co/guide/en/apm/guide/current/secret-token.html
const APM_SERVER_URL = ' *** APM Server HTTPS URL ***' // This can be found in your https://cloud.elastic.co dashboard
const ELASTICSEARCH_DB_USERNAME = '*** Elasticsearch database username here***' // Username of a user in elasticsearch database
const ELASTICSEARCH_DB_PASSWORD = '*** Elasticsearch database password here***' // Password of the username that you have set earlier

// Start Elastic APM - METRICS Collection
const apm = require('elastic-apm-node').start({
  serviceNodeName: APM_SERVICE_NODE_NAME,
  serviceName: APM_SERVICE_NAME,
  secretToken: APM_SERVER_SECRET_TOKEN,
  serverUrl: APM_SERVER_URL
})

// Start Elastic APM - LOGGING Collection
const elasticApmLogger = require('elastic-apm-logger')
elasticApmLogger.startLogging({
  esAuthObject: {
    node: ELASTICSEARCH_DB_URL,
    auth: {
      username: ELASTICSEARCH_DB_USERNAME,
      password: ELASTICSEARCH_DB_PASSWORD
    }
  },
  serviceName: APM_SERVICE_NAME,
  apmObject: apm
})

ECS (Elastic Common Schema) Formatters

You can also format your logs into ECS format before sending it back using popular NodeJS logging libraries like morgan, pino or winston - https://www.elastic.co/guide/en/ecs-logging/nodejs/current/intro.html

The elastic-apm-logger library will then detects if the logs are in ECS formatted JSON and send those back accordingly, no other configuration required!

Drawbacks

This library inadvertably creates an Elasticsearch "Dependency" icon on your "Service Maps" view, as it sends your logs back directly to your Elasticsearch Database instance.