simple-event-sourcing

Embedded json event sourcing to/from a file structure.

Quick Start

Install the dependency:

npm i @mountainpass/simple-event-sourcing

Create an event source, a consumer and write some events:

import { EventConsumer, FileEventSource } from '@mountainpass/simple-event-sourcing'

// create the event source, and wait for it to catch up
const es = new FileEventSource(
  'bare-minimum' // <- event source name (must be unique)
)
await es.hasCaughtUp() // <- required

// create the event consumer
const consumer = new EventConsumer(
  '/tmp/bare-minimum-snapshot.json', // <- state backup file
  {}, // <- configuration
  {}, // <- defaultState
  (event, index: number) => {} // <- event handler
)

// add the consumer to the event source, and wait for it to catch up
es.subscribe(consumer)
await consumer.hasCaughtUp() // <- required

// then write some events
for (let i = 0; i < 20; i++) es.store({ ts: Date.now(), name: 'event' + i })

Further Usage

Please see the examples folder for usage.

Architecture

EventWatcher (not provided) - writes events to the event source

    ⇩

EventSource (base class provided) - receives and stores events, synchronises events to multiple consumers.

    ⇩

EventConsumers (base class provided) - receives ordered events, maintains Index position and persisted State.

FAQs

• Why does the event source need to wait for hasCaughtUp() ?

  The hasCaughtUp() promise resolves once the event source has read in all data from the disk cache. This allows it to know where the HEAD pointer index is, and where it needs to write to for new events.

• Why does the consumer need to wait for hasCaughtUp() ?

  Once added to an event source, the consumer will attept to read any new events from the event source, that have occurred since it's last synchronisation.

NOTE: Changes are coming soon, which will negate the need to wait for the event source and consumer to wait until caught up.

This will include an events model, which will fire the following events:

• initialised - once all cached events have been loaded
• idle - once all queued events have been handled

Goals

Verified (tested) goals are checked.

• [x] vanilla - no external libraries (except split2)
• [x] scalable - chunked sequential file storage for optimised file access and backups
• [x] usable - support for Typescript types
• [x] asynchronous - support for Promises
• [x] streamable - support for Streams (read & write)
• [x] compressed - stores json events using a jsonl & gzip format
• [x] configurable - can be configured via defaults / environment / constructor
• [x] documented - should have plenty of javadocs and README.md examples
• [x] performant - 100k records should be stored in under 1 second (uncompressed & compressed)
• [x] snapshots - for resumable data processing
• [x] embedded - events can be push & pulled to event consumers
• [x] simple - no source files >200 lines
• [x] low footprint - low memory overhead
• [ ] resilient - exceptions don't crash the system
• [ ] verifiable - for state checking

Optimisations

• [x] don't do a json parse on unwanted lines
• [ ] create a custom readable stream to handle back pressure and knowing what's yet to be written
• [ ] create a custom writable stream to automatically handle writing to output files
• [ ] stream readahead - preload files into memory
• [ ] skip reading in files based on the desired index (requires knowledge of chunkSize, OR store fileIndex in snapshot OR ES stores own index)
• [ ] only read through files once - when initialising all consumers at same time (consumers should dictate what needs to be read in, and nothing more)
• [ ] cache last X events in memory
• [ ] absorb external library split2 ?
• [ ] verify gz files are readable on write, before deleting
• [ ] Address "TODO" comments

Behaviour Questions

• [ ] what happens when an event source receives events, before it's caught up?
  • it should be able to receive events at any time, cache the stored events, and write them once caught up.
• [ ] what happens when a consumer receives events, before it's caught up?
  • it should be able to receive events at any time, cache the stored events, and handle them once caught up.