⚡️ Serverless Kinesis Management Plugin

Example Single Managed Stream

About the plugin

HIGHLY EXPERIMENTAL AND WILL CHANGE. We are building this strictly to meet our own needs, but are willing to consider and accommodate others if it betters the overall plugin!

This plugin serves to simplify the creation and management of Kinesis streams in AWS. All the configuration is done via CloudFormation as part of the normal package step of the Serverless Framework so you can easily review all changes before the take place.

A core goal is to limit the redundant task of creating the CloudFormation for a kinesis stream, a Firehose to archive it, etc.

Limits

Each particular stream can support around 5 lambda subscribers before throttling becomes a large issue.

NOTE: If you are using the archive option of this plugin, you are limited to FOUR (4) lambda subscribers as the Kinesis Firehose will take one of your five slots.

Configuration

# serverless.yml

# Example Configs
  custom:
    kinesisStreams:
      defaults: # optional global overrides
        archiveBucket: # default: randomly named by cfn
        archive: false
        encryption: true
        encryptionKey: alias/aws/kms
        retention: 24
        shardCount: 1
        archiveTransformNewlines: false
      streams:
        - name: MyStream (required)
          archiveBucket: # optional (and currently unsupported)
          archive: false # optional
          keyId: alias/aws/kinesis # optional
          retention: 24 # optional
          shardCount: 1 # optional
          archiveTransformNewlines: false # optional

Future

In the future we hope to automate more of the operations, including intelligent auto-scaling of shards and potentially managed replication of streams as the subscriber interest grows against a particular stream.