AWS EMR Serverless SFN Waiter Workflow

This CDK library demonstrates how to implement a Step Function workflow that starts an EMR Serverless job and waits until the job completes before continuing the workflow execution.

Workflow

The workflow steps are as flows:

1. Call the StartJobRun EMR Serverless API.
2. Call GetJobRun EMR Serverless API to get the job run status.
3. Depending on the job run status do the following:
   1. If the status is SUCCESS, proceed to the success state chain.
   2. IF the status is either FAILED or CANCELLED, proceed to the fail state chain.
   3. Otherwise, wait for 60 seconds and retry the status check.

Usage:

The library contains a helper function chainEmrJobWaitPattern that can be used from any CDK construct. Given an emrServerlessApp of type CfnApplication the helper function can be used in the following manner:

const runJobState = new CallAwsService(this, "RunSparkJob", {
    service: "emrserverless",
    action: "startJobRun",
    resultPath: "$.JobInfo",
    iamResources: [emrServerlessApp.attrArn],
    parameters: {
        ApplicationId: emrServerlessApp.attrApplicationId,
        "ClientToken.$": "States.UUID()",
        JobDriver: {
            SparkSubmit: {
                EntryPoint: "s3://example-jar/job.jar",
                SparkSubmitParameters: "MainClass",
            },
        },
        ExecutionRoleArn: jobRole.roleArn,
    },
});

const successState = new Succeed(this, "SuccessState");
const failState = new Fail(this, "FailState");

// Create a workflow that polls for job completion before continuing to either success or fail chains.
const definition = chainEmrJobWaitPattern(this, emrServerlessApp, runJobState, successState, failState);