ÆGIS formerly MicroLib

N.B. This is the host - see the library here.

| Repo | Edit with GitPod | Edit with VSCode | | -------------| -------------------|-------------------| Host | | | | Library | | |

Federated Microservices

• loaded from multiple network locations and repositories at runtime;
• developed by multiple teams;
• run together in a single process;
• can be reloaded at any time without restarting the process or interrupting other components running in the process;
• are not installed on the server but streamed over the network as needed.

TL;DR

git clone https://github.com/module-federation/aegis-host
cd aegis-host
cp dotenv.example .env
yarn
yarn build
yarn start
yarn demo

Note: you no longer need to run the MicroLib-Example project, as the host has been configured to stream the federated modules directly from GitHub.

Importing your own app

Purpose

Stop paying the "microservices premium."

When evaluating microservices as a candidate architecture, the most important fact to consider is that you are building a distributed application. Microservices are the components of distributed applications - and distribution is what enables their chief virtue, deployment independence. Unfortunately, relative to the traditional alternative, monoliths, distributed apps are much harder to build and manage. So much so, that many microservice implementations fail.

This trade-off, dealing with the increased scope, cost and risk that stems from distribution, is called paying the "microservices premium". Sometimes the premium is well worth it. But in many cases it does more harm than good, leading many experts to advise against starting with microservices, but to instead introduce them gradually as scope or demand increases.

That said, in cases where the implementation does succeed, organizations generally prefer microservices to monoliths because of the increased speed and agility that deployment independence brings. So one could make the argument that if the premium were somehow discounted, microservices would be appropriate for a much wider audience.

Consider, then, what would happen if we could eliminate the need for distribution and still allow for independent deployment.

Is there such an alternative? Fowler describes the implicit premise behind the distribution/deployment trade-off:

"One main reason for using services as components (rather than libraries) is that services are independently deployable. If you have an application that consists of multiple libraries in a single process, a change to any single component results in having to redeploy the entire application.”

While technologies that support hot deployment have been around for some time (such as OSGi), it would appear, up until now anyway, perhaps due to complexity, labor intensity, or skills scarcity, they haven't been considered a viable option. Whatever the reason, with the advent of module federation, this view is no longer warranted.

Using module federation, it is possible to dynamically and efficiently import remote libraries, just as if they had been installed locally, with only a few, simple configuration steps. MicroLib exploits this technology to support a framework for building application components as independently deployable libraries, call them microservice libraries.

Using webpack dependency graphs, code splitting and code streaming, MicroLib supports hot deployment of federated modules, as well as any dependencies not present on the host, allowing development teams to deploy whenever they choose, without disrupting other components, and without having to coordinate with other teams. To simplify integration, promote composability and ensure components remain decoupled, MicroLib implements the port-adapter paradigm from hexagonal architecture to standardize the way modules communicate, so intra- and interprocess communication is transparent. E.g. whether deployed locally to the same MicroLib host instance or remotely, its all the same to the module developer.

With MicroLib, then, you get the best of both worlds. You are no longer forced to choose between manageability and autonomy. Rather, you avoid the microservices premium altogether by building truly modular and independently deployable component libraries that run together in the same process (or cluster of processes), in what you might call a "polylith" - a monolith comprised of multiple (what would otherwise be) microservices.

Features

The goal of MicroLib is to provide an alternative to distributed systems and the performance and operational challenges that come with them, while preserving the benefits of deployment independence. To this end, MicroLib organizes components according to hexagonal architecture, such that the boundaries of, and relations between, federated components are clear and useful.

In addtion to zero-install, hot deployment and local eventing, MicroLib promotes strong boundaries between, and prevents coupling of, collocated components through the formalism of the port-adapter paradigm and the use of code generation to automate boilerplate integration tasks. Features include:

• Dynamic API generation for federated modules
• Dynamic, independent persistence of federated modules
• Runtime port generation
• Runtime port-adapter binding
• Runtime adapter-service binding
• Runtime dynamic linking of WebAssembly ports, adapters and service modules
• Hot deployment of federated modules
• Configuration-based service integration
• Configuration-based service orchestration
• Built-in error handling (circuit breaker, undo)
• Common broker for locally shared (in-memory) events
• Persistence API for cached datasources
• Datasource relations for federated schemas and objects
• Object broker for retrieving external model instances
• Dependency/control inversion (IoC)
• Zero downtime, "zero install" deployment
• Evergreen deployment and semantic versioning
• Dynamic A/B testing
• Host runs as a Server, Cluster or Severless Function
• Vendor-agnostic serverless deployment (no vendor lock-in)
• Fast deployment - no-install deployment is the shortest path
• Self-deployment - built-in deployment automation
• Configurable serialization for network and storage I/O
• Clustering for availability and scalibilty
• Cluster cache synchronization
• Polyrepo code reuse (the answer to the shared code question)
• Automatated CA certifcate setup and renewal
• Built-in, pluggable service mesh
• Support for WebAssembly modules as models, adapters, services
• WebAssembly workflow - pipe modules togther to form control flows
• Polyglossia - write components in any lang with a WASM compile target
• Eventually MLOps - ci/cd pipeline for machine learning deployment
• Sooner than later AIOps - deep learning for adaptive, lights-outs operations

Components

MicroLib uses a modified version of Webpack Module Federation to import remote modules over the network into the host framework at runtime. MicroLib modules fall into three categories: model, adapter and service.

A model is a domain entity/service - or in polylith architecture, a component - that implements all or part of the service’s core logic. It also implements the MicroLib ModelSpecification interface. The interface has many options but only a few simple requirements, so developers can use as much, or as little, of the framework's capabilities as they choose.

One such capability is port generation. In a hexagonal or port-adapter architecture, ports handle I/O between the application and domain layers. An adapter implements the port ’s interface, facilitating communication with the outside world. As a property of models, ports are configurable and can be hot-added, -replaced or -removed, in which case the framework automatically rebinds their adapters as needed. Adapters by themselves can also be hot-replaced and rebound.

A service provides an optional layer of abstraction for adapters and usually implements a client library. When an adapter is written to satisfy a common integration pattern, a service implements a particular instance of that pattern, binding to the outside-facing end of the adapter. Like adapters to ports, the framework dynamically imports and binds services to adapters at runtime.

Persistence

The framework automatically persists domain models as JSON documents using the default adapter configured for the server. In-memory, filesystem, and MongoDB adapters are provided. Adapters can be extended and individualized per model. Additionally, de/serialization can be customized. Finally, every write operation generates an event that can be forwarded to an external event or data source.

A common datasource factory manages adapters and provides access to each service’s individual datasource. The factory supports federated schemas (think GraphQL) through relations defined between datasources in the ModelSpec. With local caching, not only are data federated, but so are related domain models.

const customer = order.customer(); // relation `customer` defined in ModelSpec

const creditCard = customer.decrypt().creditCardNumber;

Access to data and objects requires explicit permission, otherwise services cannot access one another’s code or data. Queries execute against an in-memory copy of the data. Datasources leverage this cache by extending the in-memory adapter.

Integration

Ports & Adapters

When ports are configured in the ModelSpecification, the framework dynamically generates methods on the domain model to invoke them. Each port is assigned an adapter, which either invokes the port (inbound) or is invoked by it (outbound).

Ports can be instrumented for exceptions and timeouts to extend the framework’s circuit breaker, retry and compensation logic. They can also be piped together in control flows by specifying the output event of one port as the input or triggering event of another.

An adapter either implements an external interface or exposes an interface for external clients to consume. On the port end, an adapter always implements the port interface; never the other way around. Ports are a function of the domain logic, which is orthogonal to environmental concerns.

Ports optionally specify a callback to process data received on the port before control is returned to the caller. The callback is passed as an argument to the port function. Ports can be configured to run on receipt of an event, API request, or called directly from code.

Ports also have an undo callback for implementing compensating logic. The framework remembers the order in which ports are invoked and runs the undo callback of each port in reverse order, starting at the point of failure. This allows transactions across multiple services to be rolled back.

Local & Remote Events

In addition to in-memory function calls, federated objects and ports, services can communicate with one another locally the same way they do remotely: by publishing and subscribing to events. Using local events, microservice libraries are virtually as decoupled as they would be running remotely.

The framework provides a common broker for local service events and injects pub/sub functions into each model:

modelA.listen(event, callback);

modelB.notify(event, data);

Local events can also be forwarded to remote event targets. Like any external integration remote ports must be configured for external event sources/sinks. Adapters are provided for Kafka and WebSockets.

Orchestration

Service orchestration is built on the framework’s port-adapter implementation. As mentioned, ports both produce and consume events, allowing them to be piped together in control flows by specifying the output event of one port as the input event of another. Because events are shared internally and can be forwarded externally, this implementation works equally well whether services are local or remote.

Callbacks specified for ports in the ModelSpec can process data received on a port before its output event is fired and the next port runs. If not specified, the framework nevertheless saves the port output to the model. Of course, you can implement your own event handlers or adapter logic to customize the flow.

Service Mesh

Aegis provides an in-process service mesh that ties Aegis instances together in an application fabric, where data, code and workload can be distributed and deployed dynamically in response to functional or non-functional requirements and conditions. The service mesh is built into federation host and in some implementations runs on the same port (e.g. Websockets version). It is not a sidecar, but external clients can connect to the mesh (e.g. to request status, observe traffic, etc). The mesh enables transparent integration of aegis components, such that component developers can write business logic that is valid regardless of where components are deployed. The service mesh is pluggable. It is straightforward to plug in another implementation. Default implementation is "webswitch" a switched mesh based on websockets. Nats and a QUIC-based implementation are planned.

See http://localhost/aegis.config.json

public/aegis.config.json

Running the Application

See above TL;DS section for a simplied install. Get up and running in about 60 seconds.

Datasources

In the default configuaton, aegis uses the local filesystem for default persistence. Alternatively, you can install MongoDB and update the .env accordingly to change the default to Mongo. You can also update an individual model's datasource in the ModelSpec.

brew install mongodb-community
mongod

.env

DATASOURCE_ADAPTER=DataSourceMongoDb
MONGODB_URL=mongodb://localhost:27017

Clustering

Aegis supports clustering with automatic cache synchronization and rolling restart for increased stability, scalality and efficiency with zero downtime. To enable:

.env

CLUSTER_ENABLED=true

Serverless

Alternatively, Aegis can run as a serverless function. It's rated for AWS. Support can be extended to other platforms and vendors by writing a message parser that simply amaps the input and output to request and response objects, indicating the HTTP method. See /adapters/serverless/parsers

Authorization

ÆGIS supports JSON Web Tokens for authorization of protected routes. To enable, you must provide JSON Web Key URI to retrieve the public key of the signer of the JSON Web Token. You can set up an account with Auth0 for testing purposes. You update the key set configuration in the public/aegis.config.json file.

public/aegis.config.json

{
  "cache": true,
  "rateLimit": true,
  "jwksRequestsPerMinute": 5,
  "jwksUri": "https://dev-2fe2iar6.us.auth0.com/.well-known/jwks.json",
  "audience": "https://microlib.io/",
  "issuer": "https://dev-2fe2iar6.us.auth0.com/",
  "algorithms": ["RS256"]
}

.env

AUTH_ENABLED=true

Transport Layer Security (HTTPS)

When ÆGIS starts (in server mode), it will check for the presence of certificate.pem and privatekey.pem files in the cert folder. If not there, it will automatically provision an x509 certificate for your domain using the ACME standard and write the files to the cert directory. The following environment must be set. Note: if NODE_ENV is set to anything other than prod the systems will provision a test certificate.

.env

NODE_ENV=prod
DOMAIN=aegis.module-federation.org
SSL_ENABLED=true

Importing your own repo

Two options are available: EASY BUTTON or DIY.

EASY BUTTON

Click here to generate the scaffolding for building a federated application with Aegis.

DIY

To import your own models, update the webpack/remote-entries.js to point to your remoteEntry.js file and change owner, repo, filedir, and branch accordingly, if using GitHub as a remote. You must specify these same attributes in your repo, only in webpack.config.js publicPath as URL params. Also from microlib-example, you'll need the same version of webpack and the extensions in the webpack dir.

Installation

Zero Downtime - Zero Install Deployment, API Generation

Reference Architecture

MicroLib prevents vendor lock-in by providing a layer of abstraction on top of vendor serverless frameworks. A vendors API gateway simply proxies requests to the MicroLib serverless function, which is the only function adapted to the vendor's platform. From that point on, MicroLib handles the "deployment" of functions as federated modules. Developers don't even need to know what cloud is hosting their software!

Further Reading

Federated Microservices: Manageable Autonomy

Stop Paying the Microservice Premium: Eliminating the Microservices Deployment/Distribution Trade-Off

Federated Applications: E Plurbis Unum

Self-Deploying Applications: Deployment Automation That Works Anywhere

Cell-based Architecture and Federation

Clean Micoservices: Building Composable Microservices with Module Federation

Webpack 5 Module Federation: A game-changer in JavaScript architecture

Microservice trade-offs

Microservice Library Videos