Core Library for xkite, a Kafka Integrated Testing Environment

xkite-core provides a comprehensive prototyping, testing, and monitoring toolset for Apache Kafka. Use xkite to bootstrap your next project, or install our library into an existing project. Built by developers, for developers.

Dependencies

The latest stable versions of:

• Node.js and NPM
• docker-compose

Installation

1. Clone this Repository git clone https://github.com/oslabs-beta/xkite-core.git

2. Install Dependencies cd into the cloned repository and run npm install

Who Uses xkite-core

• xkite GUI
• xkite CLI

How It Works

The xkite-core library is, as the name suggests, the core library for xkite.

To interface with xkite-core, simply import the Kite class into your project.

In xkite, Kite provides the underlying functionality for configuring a Docker Compose YAML configuration, managing docker containers (configure, run, pause and shutdown), interfacing with remote xkite servers, and providing configuration settings for developers to easily connect to Kafka instances.

Kite Class Data Types

Click to expand details.

Overall state of Kite, provided by Kite.getState().

type KiteState =
  | 'Unknown'
  | 'Init'
  | 'Configured'
  | 'Running'
  | 'Paused'
  | 'Shutdown';

State of remote Kite connection, provided by Kite.getServerState().

type KiteServerState = 'Disconnected' | 'Connected';

Input object to create a docker instances. Used for Kite.configure().

interface KiteConfig {
  kafka: KiteKafkaCfg;
  db?: dbCfg;
  sink?: sinkCfg;
  grafana?: grafanaCfg;
  prometheus?: prometheusCfg;
}

Response object from Kite.getSetup() available after Kite is configured.

interface KiteSetup {
  dBSetup?: dbCfg;
  kafkaSetup: KafkaSetup;
  spring?: { port: number };
  prometheus?: { port: number };
  grafana?: { port: number };
  zookeeper?: { ports: number[] };
  jmx?: { ports: number[] };
  jupyter?: { port: number };
  spark?: { port: number[] };
  docker?: { services: string[] };
}

Response object from Kite.getKafkaSetup() available after Kite is configured.

interface KafkaSetup {
  clientId: string;
  brokers: Array<string>;
  ssl?: boolean;
}

Format of the configuration file object provided by Kite.getConfigFile().

Note: the fileStream object is a stream of the docker-compose.yml file generated from Kite.configure()

interface KiteConfigFile {
  header?: any;
  fileStream: Buffer;
}

Configuration object as a part of the KiteConfig object. It defines which data source the user wants configured.

interface dbCfg {
  name: 'postgresql' | 'ksql';
  port?: number | undefined;
  postgresql?: {
    username: string;
    password: string;
    dbname: string;
  };
  ksql?: {
    schema_port?: number | undefined;
  };
  kafkaconnect?: {
    port?: number | undefined;
  };
}

Configuration object as a part of the KiteConfig object. It defines which data sink the user wants configured

interface sinkCfg {
  name: 'jupyter' | 'spark';
  port?: number;
  rpc_port?: number;
  kafkaconnect?: {
    port?: number | undefined;
  };
}

Configuration object as a part of the KiteConfig object. It defines which port the user wants their grafana interface to be configured on

interface grafanaCfg {
  port?: number | undefined;
}

Configuration object as a part of the KiteConfig object. It defines prometheus settings the user wants configured such as port, scrape and evaluation intervals

interface prometheusCfg {
  port?: number | undefined;
  scrape_interval?: number; //seconds
  evaluation_interval?: number; //seconds
}

Kite Class Methods

Click to expand details.

Note: If no input is give a default configuration will be used.

Type Definition:

configure: (arg?: string | KiteConfig | undefined) => Promise<'void'>;

Example:

const { Kite } = require('xkite-core');

await Kite.configure(); // configure local default
// or
await Kite.configure('http://localhost:3000'); // configure remote
// or

deploys all configured docker instances from Kite.configure(). If the Kite serverState === "Connected" then this deployment will happen on the remote server.

Type Definition:

deploy: () => Promise<void>;

Example:

const { Kite } = require('xkite-core');

await Kite.deploy();

pauses any/all running docker instances. If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

pause: (service?: string[] | undefined) => Promise<any>;

Example:

const { Kite } = require('xkite-core');

await Kite.pause(['kafka1', 'kafka2']); // pauses kafka1 and kafka2 docker services

await Kite.pause(); // pauses all docker instances

Unpauses any/all running docker instances. If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

unpause: (service?: string[] | undefined) => Promise<any>;

Example:

const { Kite } = require('xkite-core');

await Kite.unpause(['kafka1', 'kafka2']); // unpauses kafka1 and kafka2 docker services

await Kite.unpause(); // unpauses all docker instances

Shuts down all running or paused docker instances and removes all configured volumes. If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

shutdown: () => Promise<any>;

Example:

const { Kite } = require('xkite-core');

await Kite.shutdown();

Retrieves the KiteSetup object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

getSetup: () => KiteSetup | Promise<KiteSetup>;

Example:

const { Kite } = require('xkite-core');

const setup = await Kite.getSetup();

Retrieves the KafkaSetup object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

getSetup: () => KiteSetup | Promise<KiteSetup>;

Example:

const { Kafka } = require('kafkajs')
const { Kite } = require('xkite-core');
const kafkaSetup = await Kite.getKafkaSetup();

const kafka = new Kafka({
  ...kafkaSetup,
  clientId: 'myapp'
})
...

Retrieves the dBCfg object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

getDBSetup: () => dbCfg | Promise<dbCfg | undefined>;

Example:

const { Kite } = require('xkite-core');
const dBSetup = await Kite.getDBSetup();

Retrieves the KiteConfig object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

getConfig: () => KiteConfig | Promise<KiteConfig>;

Example:

const { Kite } = require('xkite-core');
const config = await Kite.getConfig();

Retrieves the KiteConfig object created after Kite.configure(). If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

getConfig: () => KiteConfig | Promise<KiteConfig>;

Example:

const { Kite } = require('xkite-core');
const config = await Kite.getConfig();

Retrieves the current KiteState. If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

getKiteState: () => KiteState | Promise<KiteState>;

Example:

const { Kite } = require('xkite-core');
const state = await Kite.getState();

Retrieves the state of remote connection with the Kite server.

Type Definition:

getKiteServerState: () => KiteServerState;

Example:

const { Kite } = require('xkite-core');
const serverState = await Kite.getServerState();

Retrieves the current package.zip file from Kite. This file contains the full set of dependencies to replicate the docker ecosystem sans xkite-core. If the Kite serverState === "Connected" then this command will be initiated on the remote server.

Type Definition:

getPackageBuild: () => Promise<KiteConfigFile | Error>;

Example:

const { Kite } = require('xkite-core');
const fs = require('fs');

const pkg = await Kite.getPackageBuild();
fs.writeFileSync(
        path.resolve(__dirname, 'package.zip'),
        Buffer.from(pkg.fileStream)
      );

Docker Images

xkite uses the following docker images to provide their associated services:

• xkite/kafka-connector
• bitnami/jmx-exporter
• confluentinc/cp-kafka
• confluentinc/cp-zookeeper
• prom/prometheus
• grafana/grafana-oss
• postgres
• confluentinc/ksqldb-server
• confluentinc/ksqldb-cli
• confluentinc/cp-schema-registry
• jupyterhub/jupyterhub
• bitnami/spark
• eclipse-temurin