Orchestrator Plugin for Backstage

The Orchestrator for Backstage is a mechanism designed to facilitate the implementation and execution of developer self-service flows. It serves as a vital component that enhances and augments the existing scaffolder functionality of Backstage with a more flexible and powerful set of features including long-running and asynchronous flows.

The orchestrator works harmoniously with other Backstage components such as the Software Catalog, permissions, and plugins as well as others. By leveraging its capabilities, organizations can orchestrate and coordinate developer self-service flows effectively.

Context

The Backstage Orchestrator plugin aims to provide a better option to Scaffolder, based on workflows to have a more flexible and powerful tool that addresses the need by streamlining and automating processes, allowing developers to focus more on coding and innovation.

The orchestrator relies on SonataFlow, a powerful tool for building cloud-native workflow applications.

The main idea is to keep the same user experience for users, levering the UI components, input forms, and flow that Scaffolder provides, this way it should be straightforward for users and transparent no matter whether using Templates or Workflows, both can live together being compatible with integration points.

The orchestrator controls the flow orchestrating operations/tasks that may be executed in any external service including Scaffolder Actions, this way it is possible to leverage any existing Action hence Templates and GPT can be easily migrated to workflows opening the door to extend them to more complex use cases.

Capabilities

Advanced core capabilities

• Stateful/long-lived
• Branching and parallelism
• Error management and compensation
• Event-driven supporting CloudEvents
• Audit logging
• Sub-flows
• Choreography
• Timer/timeout control
• Built-in powerful expression evaluation with JQ
• Low Code/No code
• Cloud-native architecture Kubernetes/Openshit with Operator support
• OpenAPI / REST built-in integration etc.

Client-side tooling

• Orchestration visualization / graphical editor
• Integration with service catalog/actions
• GitHub integration
• Form generation
• Runtime monitoring of instances
• Dashboards
• Potential custom integrations (user interaction, notifications, etc.)

For administrators

Installation

The Orchestrator plugin is composed of the following packages:

• @caponetto-tests/backstage-plugin-orchestrator-backend package connects the Backstage server to the Orchestrator. For setup process, see Backend Setup
• @caponetto-tests/backstage-plugin-orchestrator package contains frontend components for the Orchestrator plugin. For setup process, see Frontend Setup
• @caponetto-tests/backstage-plugin-orchestrator-common package contains shared code between the Orchestrator plugin packages.

Prerequisites

• Docker up and running (currently it is a limitation, see Limitations)

Setting up the Orchestrator as a dynamic plugin in a Helm deployment

Please follow this link for instructions: https://github.com/janus-idp/backstage-showcase/blob/main/showcase-docs/dynamic-plugins.md#helm-deployment

Setting up the configuration for the Orchestrator plugin

The following configuration is required for the Orchestrator plugin to work properly:

orchestrator:
  sonataFlowService:
    baseUrl: http://localhost
    port: 8899
    autoStart: true
    workflowsSource:
      gitRepositoryUrl: https://github.com/tiagodolphine/backstage-orchestrator-workflows
      localPath: /tmp/orchestrator/repository
  dataIndexService:
    url: http://localhost:8899

• When interacting with an existing SonataFlow infrastructure, the sonataFlowService config section must be entirely omitted and the dataIndexService.url must point to the existing Data Index Service.

For more information about the configuration options, including other optional properties, see the config.d.ts file.

• Although optional, you may also want to set up the GITHUB_TOKEN environment variable to allow the Orchestrator to access the GitHub API.

Setting up the Orchestrator backend package for the legacy backend

1. Install the Orchestrator backend plugin using the following command:

   yarn workspace backend add @caponetto-tests/backstage-plugin-orchestrator-backend

2. Create a new plugin instance in packages/backend/src/plugins/orchestrator.ts file:

   import { Router } from 'express';
   
   import { createRouter } from '@caponetto-tests/backstage-plugin-orchestrator-backend';
   
   import { PluginEnvironment } from '../types';
   
   export default async function createPlugin(
     env: PluginEnvironment,
   ): Promise<Router> {
     return await createRouter({
       config: env.config,
       logger: env.logger,
       discovery: env.discovery,
       catalogApi: env.catalogApi,
       urlReader: env.reader,
     });
   }

3. Import and plug the new instance into packages/backend/src/index.ts file:

   /* highlight-add-next-line */
   import orchestrator from './plugins/orchestrator';
   
   async function main() {
     // ...
     const createEnv = makeCreateEnv(config);
     // ...
     /* highlight-add-next-line */
     const orchestratorEnv = useHotMemoize(module, () =>
       createEnv('orchestrator'),
     );
     // ...
     const apiRouter = Router();
     // ...
     /* highlight-add-next-line */
     apiRouter.use('/orchestrator', await orchestrator(orchestratorEnv));
     // ...
   }

Setting up the Orchestrator backend package for the new backend

1. Install the Orchestrator backend plugin using the following command:

   yarn workspace backend add @caponetto-tests/backstage-plugin-orchestrator-backend

2. Add the following code to packages/backend/src/index.ts file:

   import {
     orchestratorModuleEntityProvider,
     orchestratorPlugin,
   } from '@caponetto-tests/backstage-plugin-orchestrator-backend/alpha';
   
   const backend = createBackend();
   /* highlight-add-next-line */
   backend.add(orchestratorModuleEntityProvider);
   backend.add(orchestratorPlugin);
   
   backend.start();

Setting up the Orchestrator frontend package

1. Install the Orchestrator frontend plugin using the following command:

   yarn workspace app add @caponetto-tests/backstage-plugin-orchestrator

2. Add a route to the OrchestratorPage and the customized template card component to Backstage App (packages/app/src/App.tsx):

   /* highlight-add-next-line */
   import {
     OrchestratorPage,
     OrchestratorScaffolderTemplateCard,
   } from '@caponetto-tests/backstage-plugin-orchestrator';
   
   const routes = (
     <FlatRoutes>
       {/* ... */}
       {/* highlight-add-start */}
       <Route
         path="/create"
         element={
           <ScaffolderPage
             components={{
               TemplateCardComponent: OrchestratorScaffolderTemplateCard,
             }}
           />
         }
       />
       {/* highlight-add-end */}
       {/* ... */}
       {/* highlight-add-next-line */}
       <Route path="/orchestrator" element={<OrchestratorPage />} />
     </FlatRoutes>
   );

3. Add the Orchestrator to Backstage sidebar (packages/app/src/components/Root/Root.tsx):

   /* highlight-add-next-line */
   import { OrchestratorIcon } from '@caponetto-tests/backstage-plugin-orchestrator';
   
   export const Root = ({ children }: PropsWithChildren<{}>) => (
     <SidebarPage>
       <Sidebar>
         <SidebarGroup label="Menu" icon={<MenuIcon />}>
           {/* ... */}
           {/* highlight-add-start */}
           <SidebarItem
             icon={OrchestratorIcon}
             to="orchestrator"
             text="Workflows"
           />
           {/* highlight-add-end */}
         </SidebarGroup>
         {/* ... */}
       </Sidebar>
       {children}
     </SidebarPage>
   );

For users

Using the Orchestrator plugin in Backstage

The Orchestrator plugin enhances the Backstage with the execution of developer self-service flows. It provides a graphical editor to manage workflows, and a dashboard to monitor the execution of the workflows.

Prerequisites

• Your Backstage application is installed and running.
• You have installed the Orchestrator plugin. For the installation process, see Installation.

Procedure

1. Open your Backstage application.
2. Click the Workflows tab from the left-side panel to navigate to the Orchestrator main page.
3. Inside the Orchestrator main page, you can see the list of workflows that are available in your Backstage application.