Darkspark Core Collector

This is the core HTTP request/response collection component for Darkspark. It provides a simple interface for ingesting HTTP request/response pairs, processing them locally and sending metadata to Darkspark. Metadata collected consists of:

• The method and path of the request. Certain path elements such as numbers and UUIDs are replaced with placeholder tokens.
• Query parameter keys.
• Request and response header keys.
• The request body schema.
• The response body schema.

The core collector ensures that only this metadata is collected, and the actual content of the HTTP request never leaves the local environment.

The core collector can be used in two ways:

1. As a Javascript/Typescript library
2. As a standalone executable running on Node

In either form of usage you first need a Darkspark API key. See the Darkspark Documentation to learn how to create a key.

Library

To use as a library, import darksparkCoreReceiver and HttpEventLike from darkspark-core-plug/lib, create a receiver and send it events:

const { darksparkCoreReceiver, HttpEventLike } = require('darkspark-core-plug/lib');

const receiver = darksparkCoreReceiver("<your Darkspark API key>")

const event = {
    ...
};
receiver.receive(event);

Debugging

By default the core collector operates silently. To see diagnostics and internal logging, pass a Logger implementation to darksparkCoreReceiver and enable debug mode:

const receiver = darksparkCoreReceiver(
  "<your Darkspark API key>",
  new ConsoleLogger(),
  true
);

Alternate implementations of Logger may be used to integrate with your own logging environment.

Standalone

If you are not using Javascript/Typescript, or if you prefer to hand off the Darkspark data collection to a separate process, use the core collector as a standalone executable. It can be run directly with Node, passing the API key as an argument:

node darkspark-core-plug/index.js "<your Darkspark API key>"

The process reads JSON-serialised HttpEventLike objects from stdin and forwards them to Darkspark. Each event should be written as a JSON object on a single line, followed by a newline character.

Constructing HttpEvents

The core collector normalises incoming events to HttpEvent but for convenience it also accepts HttpEventLike which has less strict requirements. In Typescript this is defined as:

export interface HttpEventLike {
  metadata: {
    schema_type: "http-event";
    schema_version: number | string;
    created_by: string;
    created_at: string;
    id: string;
  };
  request: {
    method: string;
    scheme: string;
    body?: string;
    headers: KeyValuePairs;
    host: string;
    path: string;
    port?: number | string;
    query?: KeyValuePairs;
  };
  response: {
    status_code?: number | string;
    headers?: KeyValuePairs;
    body?: string;
  };
}

It should be populated as follows:

• metadata.schema_type must be "http-event"
• metadata.schema_version must be 2
• metadata.created_by should name the source of the event
• metadata.created_at should be an ISO 8601 timestamp
• metadata.id should be a unique identifier for the event (we love UUIDs!)
• request.method is the HTTP method of the request; case is not important
• request.scheme must be either "http" or "https"
• request.body if included is a base64-encoded string of the raw request body
• request.headers is a list of KeyValuePair containing the request headers
• request.host is the target host the request was sent to
• request.path is the path component of the request URL
• request.port if included is the port the request was sent to
• request.query if included is a list of KeyValuePair containing the query parameters
• response.status_code if included is the HTTP status code of the response
• response.headers if included is a list of KeyValuePair containing the response headers
• response.body if included is a base64-encoded string of the raw response body

KeyValuePair is defined as:

export interface KeyValuePair {
  key: string;
  value: string;
}

Lists of these are used for headers and query parameters rather than objects to allow for duplicate keys.

Request and response bodies should be base64-encodings of the raw bytes of the bodies. Darkspark will interpret Content-Encoding and Content-Type headers to decode the body as necessary.