raygun-apm

raygun-apm is a module that can capture and send data about your application's performance to Raygun APM via your local Raygun Agent.

It works by using the V8 profiler API to sample the call stack while the application is running. This is the same approach used by Chrome's dev tools.

raygun-apm also supports request level profiling of web servers. This means that even though Node web applications generally run concurrently on a single thread, raygun-apm will generate traces that only include stack samples from a single request.

Setup

Node Support

The minimum supported Node versions are:

• v14.x
• v16.x*
• v18.x*

Note: v16 and v18 have not been fully tested with TCP uplink.

Additionally, the minimum supported Node versions for profiling worker threads are:

• v14.x
• v16.x

Building native dependencies (v8-profiler-node8)

Since Raygun uses a native module for V8 profiler bindings, you may need to be able to build native modules to install if a prebuilt binary is not available. We currently provide prebuilt 64-bit binaries for Windows, Linux and macOS, as well as 32-bit binaries for Windows.

• Windows: Install the windows-build-tools package (run npm install --global windows-build-tools from an Administrator PowerShell)
• macOS: Install the Xcode Command Line Tools package (run xcode-select --install) if you haven't already. You may need to install Python (>=v2.7 or >=v3.5) if using macOS 11 (Big Sur) or later.
• Unix: Ensure Python (>=v2.7 or >=v3.5), make and gcc are installed via your preferred package manager.

For more detailed information on building native modules, please consult the node-gyp documentation.

Please note, when changing Node versions, you may need to run npm rebuild v8-profiler-node8 @raygun-nickj/mmap-io --update-binary to ensure the correct versions of the native dependencies are installed.

Install package from npm

npm install raygun-apm --save

or

yarn add raygun-apm

Install and run Raygun APM Agent

To send traces to Raygun there must be a Raygun APM Agent installed and running, either locally or on another machine. If you haven't already done so, follow the instructions on the Raygun website to install and run the Agent.

If your application runs on Heroku, you can use the heroku-buildpack-raygun-apm which will run an agent on your web dynos automatically.

Usage

Profiling web server requests

raygun-apm can profile requests to node http and https servers. To enable request level profiling, add this require statement at the start of your main js file.

require('raygun-apm/http');

You can also use import if you are using TypeScript or Babel.

import 'raygun-apm/http';

raygun-apm also ships with a cross platform helper script called with-raygun-apm that will configure Node to load raygun-apm/http at startup. This can be helpful if you're working with a framework that launches your app for you.

On the command line:

node_modules/.bin/with-raygun-apm node index.js

Or in a package.json script:

"scripts": {
  "start": "with-raygun-apm node index.js"
},

The HTTP/HTTPS request profiling support works by wrapping around the node standard library http/https modules, which also means that it works out of the box with express, koa, hapi, sails, adonis, loopback, micro and most node http frameworks.

Here is an example of a simple app that uses express and node-pg:

import 'raygun-apm/http';

import express from 'express';
import {Client} from 'pg';

const app = express();

const client = new Client()
client.connect()

app.get('/', (req, res) => {
  client.query("SELECT * FROM test_data", [], (err, pgRes) => {
    res.send(JSON.stringify(pgRes.rows));
  });
});

app.listen(3030, 'localhost', () => console.log('listening on 3030'));

Capturing database queries

raygun-apm supports automatically capturing database queries from these adapters:

• pg
• mysql
• mysql2
• mssql
• mongodb
• redis
• ioredis
• @elasticsearch/elastic
• memcached
• sequelize (when used with a supported adapter)

raygun-apm will also capture incoming GraphQL queries and include them in the trace. This should work with most GraphQL servers using the official graphql package. GraphQL support has been tested and confirmed to work with:

• Apollo (apollo-server-express)
• express-graphql

require-in-the-middle is used to wrap around these libraries when they are imported.

This is necessary to detect when queries start and end, and in some cases to wrap processing of the results of a query in an asynchronous context that's associated with the original request.

When installing raygun-apm, if npm scripts are enabled for packages, raygun-apm will inspect your project's dependencies and warn you if a version of a dependency is known to be incompatible.

Please let us know if there's another database adapter or library you'd like supported.

Cloud Providers

raygun-apm can be used with a number of cloud providers.

Docker

raygun-apm can be used with Docker-based deployments. Please see the APM Agent Docker setup guide for information on installing the agent.

If the agent is running in a separate container to the Node application, you will need to set the RAYGUN_AGENT_HOST to the hostname of the agent's container.

When using docker-compose, hostnames are automatically created for each container, using the name of the service.

For example, if our APM Agent service was named raygun, as in the above example, our node application would need to have the environment variable RAYGUN_AGENT_HOST=raygun.

Heroku

raygun-apm provides support for Heroku via a custom buildpack that will install and run the APM agent alongside your application.

For more information, see the Heroku Buildpack guide on Raygun's documentation site.

AWS Elastic Beanstalk

raygun-apm provides support for AWS Elastic Beanstalk, using the .ebextensions system to install and run the APM Agent. Please see the setup guide on the Raygun documentation site for more detail.

Azure App Services

raygun-apm provides support for Azure App Services, using Azure Extensions to install and run the APM agent. Please see the APM Agent setup guide for Azure on the Raygun documentation site for instructions on installing the agent.

Performance

raygun-apm introduces a performance overhead to your application, due to the use of V8's CPU profiler and the Node async_hooks module to capture APM information. As such, using raygun-apm in production can cause noticeable slowdown to application code.

This is similar to how a web application behaves when capturing a CPU profile in Chrome. Our current benchmarks suggest this overhead is currently around 30% at worst, although we're constantly endeavouring to reduce the performance impact of raygun-apm.

Limitations

raygun-apm uses V8's CpuProfiler (via v8-profiler-node8) to sample the callstack in order to build traces. This is the same approach used by Chrome's developer tools.

V8's CpuProfiler is a sampling profiler, which means that it captures the callstack at a regular interval. By default, a callstack is captured approximately every 1 millisecond, or a thousand times a second.

As a result, there are a few limitations:

• Functions that take less than 1ms to run might not appear in the trace.
• The start/end timings for each stack frame are only accurate to ±1ms.
• As V8's JIT compiler optimizes your code at runtime, traces may appear to lose detail.
• As a result of all of the above, multiple traces for the same code may contain slightly different details.
• By default, raygun-apm communicates with the Raygun agent via UDP. As a result, some traces may fail to process due to network conditions.

It's worth noting that all of these limitations apply primarily to code that is executing so fast that it cannot be profiled without introducing a significant overhead. As such, these limitations should have minimal impact when investigating code that takes more than a few millisecond to run.

Configuration

You can set the following environment variables to control the behaviour of the profiler.

RAYGUN_AGENT_HOST

• defaults to 0.0.0.0

Used to configure the hostname for the Raygun Agent that traces will be sent to.

RAYGUN_AGENT_PORT

• defaults to 2799

Used to configure the port for the Raygun Agent that traces will be sent to.

RAYGUN_API_KEY

• not set by default
• can also be set via Raygun_ApiKey

Used to configure the key the process will use to submit traces to Raygun. This can be used to support multiple apps running on one machine using different keys. If no key is set, the Raygun agent will use the key it has been configured with.

Please note: RAYGUN_API_KEY must be set in order for an application's sampling rules to be applied. Otherwise, the APM agent will use the default sampling settings. This applies to both sampling frequency and any overrides applied via the Raygun website.

RAYGUN_APM_OVERRIDES_PATH

• not set by default
• defaults to the Raygun APM Agent overrides directory if RAYGUN_API_KEY is set
• can also be set via PROTON_USER_OVERRIDES_FILE

Used to specify a custom path to an overrides file. See the Overrides section below for details.

If you want to be able to configure your overrides via the Raygun app, ensure RAYGUN_API_KEY is set to your API key, and leave RAYGUN_APM_OVERRIDES_PATH unset.

If you want to keep your overrides in source control, set RAYGUN_APM_OVERRIDES_PATH to the location of your overrides file. This path can either be relative to the working directory of your application's processs or absolute.

RAYGUN_APM_REQUEST_FILTER_PATH

• defaults to .apmignore

Used to specify a custom path to request filter path, which can be used to control which routes are profiled when using raygun-apm/http.

See the Request Filtering section below for more details.

RAYGUN_APM_DISABLE_PROFILING

• defaults to false

If set to t or true, http servers will no longer be automatically instrumented when raygun-apm/http is loaded, and requiring raygun-apm directly will return a mock makeProfiler that does not capture profiles.

This allows for profiling to be disabled or enabled without requiring any code changes. This is useful when you want to deploy your production application with raygun-apm installed, but only enable it when investigating a performance issue.

Please node that requiring internal profile modules could still activate internal APM logic even when this option is active. As such, requiring internal profiler modules is discouraged.

RAYGUN_APM_SAMPLING_FREQUENCY

• defaults to 1000 (microseconds)

RAYGUN_APM_SAMPLING_FREQUENCY can be set to change how often stack samples are captured by V8's profiler. Setting a value lower than the default 1000 will include more detail in traces, at the expense of additional overhead. Increasing the value beyond 1000 will causes traces to contain less detail, but will reduce the performance overhead of the V8 profiler.

This variable only supports integer values.

RAYGUN_APM_CONCURRENT_PROFILE_LIMIT

• defaults to 1

This environment variable can be used to control the number of concurrent profiles that will be capturing and processing at one time. Increasing this number will capture more concurrent profiles at the cost of additional performance overhead.

RAYGUN_APM_FEATURE_LEVEL

• defaults to stable

If RAYGUN_APM_FEATURE_LEVEL is set to experimental, all feature flags for experimental features will be enabled. If unset or set to anything else, the feature level will default to stable.

RAYGUN_APM_WORKER_PROFILING

• defaults to true

By default, the worker_threads module will be patched to enable worker thread profiling. When a worker is created by code that is being profiled, or sent a message by code that is being profiled, the worker will be profiled until the main thread profile completes.

If set to f or false this patch will be disabled and workers will operate as normal without profiling support.

There should be no behavioural changes to the Worker class, although there is a small delay (1-2ms) when first starting the profile for the worker, as we ensure the profile is running before allowing the worker to receive the user's message.

The minimum Node versions needed for worker profiling support are:

• Node v14.x and onwards

This is because raygun-apm relies on a Node patch that allows transferring a separate message port on worker initialization, which we use to allow a channel of APM communications that doesn't interfere with end user messages.

By default, raygun-apm will not throw an error if the worker_threads module is already loaded, as this can cause compatibility issues with process managers like pm2.

If you encounter any issues with worker activity not appearing in traces, please confirm that raygun-apm or raygun-apm/http is required prior to requiring worker_threads or other modules that utilise Workers. For the best experience, consider using the with-raygun-apm wrapper to load raygun-apm on Node startup, as documented above.

You can also set RAYGUN_APM_WORKER_THREADS_STRICT_LOAD_ORDER=true to enable an error if the worker_threads module is loaded prior to raygun-apm.

RAYGUN_APM_PROFILE_TIMEOUT

• defaults to 60000 (60 seconds)

This environment variable controls how long a profile will capture for before timing out. This value is in milliseconds, and defaults to 60000, or 60 seconds.

If a profile is timed out, a message will be logged showing which exit points were still pending. This can help to debug long running queries or external resources, but also might be indicative of a bug in raygun-apm's exit point system. If you're unsure, please contact support.

RAYGUN_APM_MAX_QUERY_SIZE_BYTES

• defaults to 4096

This environment variable controls the maximum size of a query before it's truncated. Higher values will allow larger queries to be sent intact, but may increase packet loss and cause queries not to appear in traces.

Please note this setting only applies when using the UDP or TCP transports.

RAYGUN_APM_TCP_TRANSPORT

• defaults to false

If this environment variable is set to true, a TCP-based transport will be used instead of the default UDP implementation. This can improve the reliabliity of transmitting traces to the agent, at the cost of slightly reduced performance.

When using the TCP transport, the RAYGUN_AGENT_HOST, RAYGUN_AGENT_PORT and RAYGUN_APM_MAX_QUERY_SIZE_BYTES variables will still be respected.

Please note, in order to enable TCP support in the Raygun APM Agent, the following environment variable needs to be set when the Raygun APM Agent runs: Raygun_NetworkTransmissionMode="Tcp,Udp".

RAYGUN_APM_FILE_TRANSPORT

• defaults to false

If this environment variable is set to true, the Shared File/Shared Memory transport will be used instead of the default UDP transport. This causes raygun-apm to open a shared file and memory mapping for communication with the agent, which can be useful in environments where it's not viable to connect to the agent over UDP.

In order for traces to be submitted successfully the APM agent also needs to be running in the Shared File/Shared Memory communication mode.

This mode is automatically enabled when the agent is running on Azure, but can also be explicitly enabled by creating an appsettings.json file in the agent install directory with NetworkTransmissionMode set to File.

{
  "Agent": {
    "NetworkTransmissionMode": "File"
  }
}

Please note, this variable takes precedence over the RAYGUN_APM_TCP_TRANSPORT variable. If both are enabled or if the app is running in Azure App Services, the file-based transport will be used.

RAYGUN_APM_FILE_TRANSPORT_PATH

• not set by default

If RAYGUN_APM_FILE_TRANSPORT_PATH is enabled, this environment variable can be used to control which directory shared files will be created in. If unset, the path defaults to %ProgramData%\Raygun\CommandStream.

RAYGUN_APM_DISABLE_STACK_SIMPLIFICATION

• defaults to false

If this environment variable is set to true or t, the profiler's stack simplification logic will be disabled.

By default, stack simplification means that consecutive system frames and consecutive frames from the same library are collapsed, which helps to make traces smaller and easier to understand. If you find you need more detail in your traces around system and library behaviour, consider disabling stack simplification.

RAYGUN_APM_INCLUDE_INTERNALS

• defaults to false

If this environment variable is set to to true or t, internal Node frames, async hooks frame and frames from the profiler itself will be included in traces.

This can add a lot of noise to traces, so it's disabled by default.

RAYGUN_APM_DISABLE_ASYNC_HOOKS

• defaults to false

In order to separate the execution of multiple parallel requests into distinct traces, raygun-apm uses the async_hooks module from the Node standard library to track the asynchronous contexts used by an application and it's libraries.

async_hooks provides critical information required to implement request level profiling, but that comes at the cost of performance overhead. Depending on your Node version and the type of work an application is performing, this overhead can be significant.

The use of async_hooks can be disabled by setting RAYGUN_APM_DISABLE_ASYNC_HOOKS to true. While this will decrease performance overhead, please note that the following limitations apply:

• Traces will now contain stack samples from all requests occuring over that time span
• Database queries, http requests and worker activity may not be correctly attributed to a trace
• Traces may not be extended by timers and effects
• Exceptions from raygun4node may not be attached to traces

With that in mind, this setting is not recommended for most users. This setting is primarily useful when evaluating the tradeoff of functionality versus performance overhead that comes from using async_hooks.

Please note that the profiler will continue to use executionAsyncId and triggerAsyncId from async_hooks even when this setting is turned on, as these are populated by Node regardless of whether custom async_hooks logic has been enabled.

DEBUG

• if set to raygun-apm, detailed debug logging will be printed

Overrides

You can configure the profiler to exclude certain files and functions from the traces that are collected.

If RAYGUN_API_KEY is set, the profiler will default to looking for an overrides file in the Raygun APM Agent's data directory. This allows you to configure your overrides via the Raygun webapp.

Alternatively, you can provide an overrides file to the profiler directly by setting the RAYGUN_APM_OVERRIDES_PATH environment variable to a path pointing to an overrides file.

In your overrides files, each line should start with a - or + sign to indicate if the line should be ignored or included. The rest of the line is matched against the start of each file/function name in the trace, using a : to separate the file name and function name.

An overrides file might look like this:

# Ignore an entire directory
-node_modules

# Except for a specific sub-directory
+node_modules/pg

# Ignore a specific function
-app/utils.js:authenticate

# Ignore all but a specific function in a file
-app/middleware.js
+app/middleware.js:loadFile

This means you can provide anything from a partial path to a fully specified filename and function name.

If the path is not absolute, it will be considered to be relative to the current working directory of the process.

You can also ignore or allow Node internals, by prefixing the path with a !.

For example, to remove many of the system frames present in a typical web application, you could add these lines to your overrides file:

-!internal/stream_base_commons.js
-!_stream_readable.js
-!_stream_writeable.js
-!events.js

Request Filtering

When using raygun-apm/http, it may be desirable to prevent the profiler from running for certain routes.

This can be achieved by setting RAYGUN_APM_REQUEST_FILTER_PATH to point to a file.

Each line in the request filter file corresponds with a route that should be ignored. By default, routes will match exactly, but a * can be used as a wildcard to match many routes. Query parameters are ignored.

This example would filter out queries to /graphql or any route starting with /assets/:

/graphql
/assets/*

As with overrides, you can specify a + or - at the start of the line to indicate if the route should be included or ignored. If the line doesn't start with + or -, that route will be filtered out.

This allows for filtering out certain parts of an application's routes but still profiling some subroutes that would otherwise be ignored.

# Ignore every route inside of /admin except for /admin/editor

-/admin*
+/admin/editor

Source Maps

raygun-apm supports source maps when Node's --enable-source-maps CLI option is enabled.

Source maps are useful for applications written using languages that compile to JavaScript, like TypeScript or CoffeeScript. When source maps are available and Node's --enable-source-maps flag is enabled, raygun-apm will attempt to map the filename of user code frames to their original source file.

This means that instead of seeing the location of the generated code, such as dist/index.js, you would instead see the location of the source file, e.g. src/index.ts.

raygun-apm utilizes Node's builtin source map support, which relies on the presence of a source map annotation in the generated file.

For a TypeScript project, you can enable source map generation by setting sourceMap: true in your project's tsconfig.json file.

The minimum Node versions needed for source map support are:

• Node v14.x and onwards

API

You can profile sections of code directly by using the profiler API.

import { makeProfiler } from 'raygun-apm';

function makeProfiler(host?: string, port?: number): Profiler

Optionally takes a host and a port and returns a Profiler. If the host or port are omitted, they will default to the RAYGUN_AGENT_HOST/RAYGUN_AGENT_PORT environment variables or 0.0.0.0:2799.

Profiler

runProfile(f: (done: () => void) => void, label?: string): Promise<void>

Takes a function to profile. A done callback is passed to the profile function, which should be called when the profiler should be stopped.

const { makeProfiler } = require('.');
const http = require('http');

const profiler = makeProfiler();

profiler.runProfile((done) => {
  http.get('http://google.com', (res) => {
    res.on('error', done);
    res.on('end', done);

    res.resume();
  });
});

profiler.stop();

Returns a promise that resolves when the profile has been captured.

A label can optionally be provided as the second argument, which changes the name of the root frame used in the trace. If the string contains a ., the label will be split, and the first part will be used as the frame location, with the remaining part being used to name the frame.

stop(): Promise<void>

Stops the profiler. Closes the UDP socket and disables async_hooks.

The profiler will attempt to finish processing any already captured profiles but this is not guaranteed.

Returns a promise that resolves after the profiler has stopped.