Metamorphosis

A super dope, pluggable NodeJS Kafka Framework for building Kafka workers like consumers, producers and stream processors.

Requirements

• Kafka cluster either BYO or run the install scripts below for a test cluster
• Node 12.x
• Yarn or NPM package managers (this project defaults to Yarn)
• Docker (if testing without Kafka cluster)
• Docker Compose

Install

yarn install

Config

Copy example .env and replace values with your own

cp .env.example .env

| Option | Description | Default | | -------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------- | | KAFKA_BROKERS | Comma-delimited string of brokers e.g. 10.10.10.1:9092,10.10.10.2:9092,... | localhost:9092 | | KAFKA_CLIENT_ID | Unique ID used by consumers and producers in this app | metamorphosis | | KAFKA_TOPIC_CONSUMER | Topic to read from for a single-topic consumer app | metamorphosis.test | | KAFKA_TOPIC_PRODUCER | Topic to write to for a single-topic producer app | metamorphosis.test | | CONSUMER_GROUP_ID | Group name responsible for tracking offsets on a consumer. This should be unique for every consumer app. | metamorphosis-consumer-group |

Usage

See /examples folder for specific sample applications

Services

Allow for consistent setup processing of Kafka topics using common tools for repeated tasks.

Default Consumer

Pass a callback and perform work on each message in a single topic

Default Producer

Generate a client connection with the ability to write messages into a Kafka topic.

Webhook Producer

Define an endpoint to consume payloads and write to a Kafka topic.

Database Adapter

Pass a database adapter into any service for reading and writing to a database.

Testing

Install either the Kitchen Sink package or pick and choose from the items below.

A) Kitchen Sink

Install everything needed to test Kafka and all database adapters

yarn setup:all

B) Choose your components

You can use Kafka or database environments you are already running by setting them in the .env or the config/{environment}.js file

Kafka Cluster

Setup local Kafka cluster with Zookeper and SSL/SASL connection

yarn kafka:start

MySQL Database

Setup MySQL 8 database with sample schema and table

yarn mysql:start

Run Mocha Tests

Run all tests from root of project. You can adjust the timeout and other settings in .mocharc.json and test/config.js as well as the standard application config.

yarn test