integ-runner

The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

Overview

This tool has been created to be used initially by this repo (aws/aws-cdk). Long term the goal is for this tool to be a general tool that can be used for running CDK integration tests. We are publishing this tool so that it can be used by the community and we would love to receive feedback on use cases that the tool should support, or issues that prevent the tool from being used in your library.

This tool is meant to be used with the integ-tests library.

Usage

• Run all integration tests in test directory

integ-runner [ARGS] [TEST...]

This will look for all files that match the naming convention of /integ.*.(js|ts)$/ or /integ_*.py. Each of these files represents a self contained AWS CDK app, defining test cases and assertions using the integ-tests library.

By default, executing integ-runner will do the following for each file (app):

1. Synth each integration test to create a new snapshot in memory
2. Compare the snapshot to the previous version (i.e. files in *.snapshot/**), and fail on any differences.
   For new tests this will always fail.

To accept a snapshot update, the integreation test has to be deployed and assertions have to pass.
Execute the runner again, this time with integ-runner --update-on-fail and the following will happen for each file:

1. Deploy the previous snapshot
2. Deploy the new version as a Stack update and run assertions
3. If successful, update the snapshots

All snapshot files (i.e. *.snapshot/**) must be checked-in to version control. If not, changes cannot be compared across systems and the update workflow cannot be used.

Options

• --update-on-failed (default=false) Rerun integration tests if snapshot fails

• --clean (default=true) Destroy stacks after deploy (use --no-clean for debugging)

• --verbose (default=false) verbose logging, including integration test metrics (specify multiple times to increase verbosity)

• --parallel-regions (default=us-east-1,us-east-2, us-west-2) List of regions to run tests in. If this is provided then all tests will be run in parallel across these regions

• --directory (default=test) Search for integration tests recursively from this starting directory

• --force (default=false) Rerun integration test even if the test passes

• --profiles List of AWS Profiles to use when running tests in parallel

• --exclude (default=false) If this is set to true then the list of tests provided will be excluded

• --from-file Read the list of tests from this file

• --disable-update-workflow (default=false) If this is set to true then the update workflow will be disabled

• --app The custom CLI command that will be used to run the test files. You can include {filePath} to specify where in the command the test file path should be inserted. Example: --app="python3.8 {filePath}".

  Use together with --test-regex to fully customize how tests are run, or use with a single --language preset to change the command used for this language.

• --test-regex Detect integration test files matching this JavaScript regex pattern. If used multiple times, all files matching any one of the patterns are detected.

• --watch Run a single integration test in watch mode. In watch mode the integ-runner will not save any snapshots.

  Use together with --app to fully customize how tests are run, or use with a single --language preset to change which files are detected for this language.

• --language The language presets to use. You can discover and run tests written in multiple languages by passing this flag multiple times (--language typescript --language python). Defaults to all supported languages. Currently supported language presets are:

  • javascript:
    • File RegExp: ^integ\..*\.js$
    • App run command: node {filePath}
  • typescript:
    Note that for TypeScript files compiled to JavaScript, the JS tests will take precedence and the TS ones won't be evaluated.
    • File RegExp: ^integ\..*(?<!\.d)\.ts$
    • App run command: node -r ts-node/register {filePath}
  • python:
    • File RegExp: ^integ_.*\.py$
    • App run command: python {filePath}
  • go:
    • File RegExp: ^integ_.*\.go$
    • App run command: go run {filePath}

• --list (default=false) List tests instead of running them.

• --inspect-failures (default=false) Keep generated snapshots when differences exist in snapshot comparisons.

• --max-workers (default=16) The max number of workerpool workers to use when running integration tests concurrently.

Example:

integ-runner --update-on-failed --parallel-regions us-east-1 --parallel-regions us-east-2 --parallel-regions us-west-2 --directory ./ --language python

This will search for python integration tests recursively from the current directory and then execute them in parallel across us-east-1, us-east-2, & us-west-2.

If you are providing a list of tests to execute, either as CLI arguments or from a file, the name of the test needs to be relative to the directory. For example, if there is a test aws-iam/test/integ.policy.js and the current working directory is aws-iam you would provide integ.policy.js

integ-runner integ.policy.js

Common Workflow

A common workflow to use when running integration tests is to first run the integration tests to see if there are any snapshot differences.

integ-runner

If there are any differences you might see an output like the output below.

integ-runner

Verifying integration test snapshots...

  eks-cluster-private-endpoint No Change!
  eks-inference No Change!
  alb-controller No Change!
  eks-oidc-provider No Change!
  eks-bottlerocket-ng No Change!
  eks-cluster No Change!
  fargate-cluster No Change!
  eks-cluster-handlers-vpc No Change!
  eks-helm-asset - Snapshot changed!
Resources
[+] Custom::AWSCDK-EKS-HelmChart Clustercharttestocichart9C188967



Snapshot Results:

Tests:    1 failed, 9 total
Error: Some snapshot tests failed!
To re-run failed tests run: integ-runner --update-on-failed
    at main (packages/@aws-cdk/integ-runner/lib/cli.js:90:15)
error Command failed with exit code 1. 

To re-run the integration test for the failed tests you would then run:

integ-runner --update-on-failed

This will run the snapshot tests and collect all the failed tests. It will then re-execute the integration test for the failed tests and if successful, save the new snapshot.

integ-runner --update-on-failed

Verifying integration test snapshots...

  eks-cluster-private-endpoint No Change!
  eks-inference No Change!
  alb-controller No Change!
  eks-oidc-provider No Change!
  eks-bottlerocket-ng No Change!
  eks-cluster No Change!
  fargate-cluster No Change!
  eks-cluster-handlers-vpc No Change!
  eks-helm-asset - Snapshot changed!
Resources
[+] Custom::AWSCDK-EKS-HelmChart Clustercharttestocichart9C188967



Snapshot Results:

Tests:    1 failed, 9 total

Running integration tests for failed tests...

Running in parallel across: us-east-1, us-east-2, us-west-2
Running test test/integ.eks-helm-asset.js in us-east-1
  eks-helm-asset Test Succeeded!

Test Results:

Tests:    1 passed, 1 total

Nested stack templates are also compared as part of the snapshot. However asset hashes are ignored by default. To enable diff for asset hashes, set diffAssets: true of IntegTestProps.

Update Workflow

By default, integration tests are run with the "update workflow" enabled. This can be disabled by using the --disable-update-workflow command line option.

If an existing snapshot is being updated, the integration test runner will first deploy the existing snapshot and then perform a stack update with the new changes. This is to test for cases where an update would cause a breaking change only on a stack update.

The integ-runner will also attempt to warn you if you are making any destructive changes with a message like:

!!! This test contains destructive changes !!!
    Stack: aws-cdk-lambda-1 - Resource: MyLambdaServiceRole4539ECB6 - Impact: WILL_DESTROY
    Stack: aws-cdk-lambda-1 - Resource: AliasAliasPermissionAF30F9E8 - Impact: WILL_REPLACE
    Stack: aws-cdk-lambda-1 - Resource: AliasFunctionUrlDC6EC566 - Impact: WILL_REPLACE
    Stack: aws-cdk-lambda-1 - Resource: Aliasinvokefunctionurl4CA9917B - Impact: WILL_REPLACE
!!! If these destructive changes are necessary, please indicate this on the PR !!!

If the destructive changes are expected (and required) then please indicate this on your PR.

New tests

If you are adding a new test which creates a new snapshot then you should run that specific test with --disable-update-workflow. For example, if you are working on a new test integ.new-test.js then you would run:

integ-runner --update-on-failed --disable-update-workflow integ.new-test.js

This is because for a new test we do not need to test the update workflow (there is nothing to update).

watch

It can be useful to run an integration test in watch mode when you are iterating on a specific test.

integ-runner integ.new-test.js --watch

In watch mode the integ test will run similar to cdk deploy --watch with the addition of also displaying the assertion results. By default the output will only show the assertion results.

• To show the console output from watch run with -v
• To also stream the CloudWatch logs (i.e. cdk deploy --watch --logs) run with -vv

When running in watch mode most of the integ-runner functionality will be turned off.

• Snapshots will not be created
• Update workflow will not be run
• Stacks will not be cleaned up (you must manually clean up the stacks)
• Only a single test can be run

Once you are done iterating using watch and want to create the snapshot you can run the integ test like normal to create the snapshot and clean up the test.

cdk.context.json

cdk watch depends on a cdk.context.json file existing with a watch key. The integ-runner will create a default cdk.context.json file if one does not exist.

{
  "watch": {}
}

You can further edit this file after it is created and add additional watch fields. For example:

{
  "watch": {
    "include": ["**/*.js"]
  }
}

integ.json schema

See @aws-cdk/cloud-assembly-schema/lib/integ-tests/schema.ts

Defining an integration test

See the @aws-cdk/integ-tests module for information on how to define integration tests for the runner to exercise.

Config file

All options can be configured via the integ.config.json configuration file in the current working directory.

{
  "maxWorkers": 10,
  "parallelRegions": [
    "eu-west-1",
    "ap-southeast-2"
  ]
}

Available options can be listed by running the following command:

integ-runner --help

To use a different config file, provide the --config command-line option.