Observability

Setting up Graylog Alerts in Slack

1. Set up the Slack App in your Slack workspace for Graylog if not already done.

2. Create a new Webhook URL for your target Slack channel in the Install App section of the Slack App and copy the Webhook URL. Note that if you are just creating an additional alert for an existing channel, you do not need to create a new Webhook URL and can reuse the existing one for that channel.

3. In Graylog, under Alerts > Notifications, create a new notification and select Slack Notification. In the configuration, set the Webhook URL from Slack, specify the channel name, and customize the message format as needed. It is not currently possible to duplicate Notifications in Graylog, so you might need to manually copy the configuration if you want to create multiple similar notifications. See below for the recommended notification template.

4. In Graylog, under Alerts > Event Definitions, create a new Event Definition or edit an existing one. Recommend to select an existing Event Definition and duplicate it. Modify the search query and other properties as necessary. Be sure to explicitly create fields and specify each field name as Template with value ${source.<field_name>}. This will allow you to access the field values in the notification template. See below for the recommended fields to specify in the Event Definition. Assign the Notification that you created to the Event Definition.

5. Enable the Event Definition in the Graylog console. You can test the alert by manually triggering the alert condition in the Event Definition.

Recommended Notification Template

${if event.fields.error_name}${event.fields.error_name}: ${end}${if event.fields.error_message}${event.fields.error_message}${end}
Occurred at: ${event.timestamp}
${if event.fields.name}Occurred in: ${event.fields.name}${end}
${if event.fields.correlation}Correlation: ${event.fields.correlation}${end}
${if event.fields.operator_id}Operator: ${event.fields.operator_id}${end}
${if event.fields.error_data}${event.fields.error_data}${end}
${if event.fields.error_stack}${event.fields.error_stack}${end}
https://graylog.unconventionalcode.com/alerts/${event.id}/replay-search

Recommended Fields to Specify in the Event Definition

-- error_name (string) - The name of the error or alert (automatically set log.error) -- error_message (string) - The message of the error or alert (automatically set log.error) -- error_stack (string) - The stack trace of the error or alert (automatically set log.error) -- error_data (string) - Additional data about the error or alert (automatically set log.error) -- name (string) - The name of the service or component where the error occurred (automatically declared in logger constructor) -- correlation (string) - A unique identifier for the error or alert (automatically declared in logger constructor) -- operator_id (string) - The ID of the operator who triggered the alert (recommended to pass in every log metadata object, e.g. log.error(error, { operator_id: '1234' }))

Setting up the logger

From version 0.0.32 onward, you must include in your environment variables:

NEW_RELIC_API_KEY
LOGGING_NEW_RELIC_ENABLED
LOGGING_CONSOLE_ENABLED
LOGGING_GRAYLOG_ENABLED

If you omit any of these environment variables, the logger will not work as expected.