Bulk Import Backend Plugin

This is bulk-import-backend plugin which provides Rest API to bulk import catalog entities into the catalog

For administrators

Installation and Configuration

Setting up the bulk import backend package for the legacy backend

1. Install the NPM Package

# From your backstage root directory
yarn workspace backend add @janus-idp/backstage-plugin-bulk-import-backend

1. Create a plugin instance in the src/packages/backend/plugins/bulk-import.ts file:

import { HostDiscovery } from '@backstage/backend-common';
import { CatalogClient } from '@backstage/catalog-client';

import { Router } from 'express';

import { BulkImportApi } from '@janus-idp/backstage-plugin-bulk-import-backend';

import { PluginEnvironment } from '../types';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  const catalogApi = new CatalogClient({
    discoveryApi: env.discovery,
  });
  return await createRouter({
    config: env.config,
    logger: env.logger,
    permissions: env.permissions,
    catalogApi: env.catalogApi,
  });
}

1. Import the plugin into the backend in the packages/backend/src/index.ts file:

   /* highlight-add-next-line */
   import bulkImport from './plugins/bulk-import';
   
   async function main() {
     // ...
     const createEnv = makeCreateEnv(config);
     // ...
     /* highlight-add-next-line */
     const bulkImportEnv = useHotMemoize(module, () => createEnv('bulkImport'));
     // ...
     const apiRouter = Router();
     // ...
     /* highlight-add-next-line */
     apiRouter.use('/bulk-import', await bulkImport(bulkImportEnv));
     // ...
   }

Setting up the bulk import backend package for the new backend

1. Install the bulk import backend plugin using the following command:

   yarn workspace backend add @janus-idp/backstage-plugin-bulk-import-backend

2. Add the following code to the packages/backend/src/index.ts file:

   const backend = createBackend();
   /* highlight-add-next-line */
   backend.add(import('@janus-idp/backstage-plugin-bulk-import-backend/alpha'));
   
   backend.start();

Permission Framework Support

The Bulk Import Backend plugin has support for the permission framework. A basic example permission policy is shown below to disallow access to the bulk import API for all users except those in the backstage-admins group.

1. Create a backend module for the permission policy, under a packages/backend/src/plugins/permissions.ts file:

import { createBackendModule } from '@backstage/backend-plugin-api';
import { BackstageIdentityResponse } from '@backstage/plugin-auth-node';
import {
  AuthorizeResult,
  isPermission,
  PolicyDecision,
} from '@backstage/plugin-permission-common';
import {
  PermissionPolicy,
  PolicyQuery,
} from '@backstage/plugin-permission-node';
import { policyExtensionPoint } from '@backstage/plugin-permission-node/alpha';

import { bulkImportPermission } from '@janus-idp/backstage-plugin-bulk-import-common';

class BulkImportPermissionPolicy implements PermissionPolicy {
  async handle(
    request: PolicyQuery,
    user?: BackstageIdentityResponse,
  ): Promise<PolicyDecision> {
    if (isPermission(request.permission, bulkImportPermission)) {
      if (
        user?.identity.ownershipEntityRefs.includes(
          'group:default/backstage-admins',
        )
      ) {
        return { result: AuthorizeResult.ALLOW };
      }
    }
    return { result: AuthorizeResult.DENY };
  }
}

export const BulkImportPermissionBackendModule = createBackendModule({
  pluginId: 'permission',
  moduleId: 'custom-policy',
  register(reg) {
    reg.registerInit({
      deps: { policy: policyExtensionPoint },
      async init({ policy }) {
        policy.setPolicy(new BulkImportPermissionPolicy());
      },
    });
  },
});

2. Import @backstage/plugin-permission-backend/alpha and add your permission module to the packages/backend/src/index.ts file:

import { BulkImportPermissionBackendModule } from './plugins/permissions';

backend.add(BulkImportPermissionBackendModule);
backend.add(import('@backstage/plugin-permission-backend/alpha'));

Audit Logging

Audit logging is backed by the backstage-plugin-audit-log-node package. The Bulk Import Backend plugin adds the following events to the backend audit logs:

• BulkImportUnknownEndpoint: tracks requests to unknown endpoints.

• BulkImportPing: tracks GET requests to the /ping endpoint, which allows to make sure the bulk import backend is up and running.

• BulkImportFindAllOrganizations: tracks GET requests to the /organizations endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.

• BulkImportFindRepositoriesByOrganization: tracks GET requests to the /organizations/:orgName/repositories endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).

• BulkImportFindAllRepositories: tracks GET requests to the /repositories endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.

• BulkImportFindAllImports: tracks GET requests to the /imports endpoint, which returns the list of existing import jobs along with their statuses.

• BulkImportCreateImportJobs: tracks POST requests to the /imports endpoint, which allows to submit requests to bulk-import one or many repositories into the Backstage Catalog, by eventually creating import Pull Requests in the target repositories.

• BulkImportFindImportStatusByRepo: tracks GET requests to the /import/by-repo endpoint, which fetches details about the import job for the specified repository.

• BulkImportDeleteImportByRepo: tracks DELETE requests to the /import/by-repo endpoint, which deletes any existing import job for the specified repository, by closing any open import Pull Request that could have been created.

Example:

{
  "actor": {
    "actorId": "user:default/myuser",
    "hostname": "localhost",
    "ip": "::1",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
  },
  "eventName": "BulkImportFindAllOrganizations",
  "isAuditLog": true,
  "level": "info",
  "message": "'get /organizations' endpoint hit by user:default/myuser",
  "meta": {},
  "plugin": "bulk-import",
  "request": {
    "body": {},
    "method": "GET",
    "params": {},
    "query": {
      "pagePerIntegration": "1",
      "sizePerIntegration": "5"
    },
    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
  },
  "response": {
    "status": 200
  },
  "service": "backstage",
  "stage": "completion",
  "status": "succeeded",
  "timestamp": "2024-08-26 16:41:02"
}

For Users

Usage

The bulk import backend plugin provides a REST API to bulk import catalog entities into the catalog. The API is available at the /api/bulk-import endpoint.

As a prerequisite, you need to add at least one GitHub Integration (using either a GitHub token or a GitHub App or both) in your app-config YAML file (or a local app-config.local.yaml file). See https://backstage.io/docs/integrations/github/locations/#configuration and https://backstage.io/docs/integrations/github/github-apps/#including-in-integrations-config for more details.

REST API

Please refer to src/schema/openapi.yaml for the API definition (along with some examples) and the generated documentation for more details about the request and response parameters and formats.