CI/CD tool for Azure Stream Analytics jobs

• CI/CD tool for Azure Stream Analytics jobs
  • Install
  • Build project
  • Configure autoscale settings
  • Run locally
  • Add automated test
    • Add a test case
    • Run unit test
  • Deploy to Azure
  • Next steps

The Azure Stream Analytics (ASA) CI/CD npm package allows you to automatically build, test, and deploy your Stream Analytics projects. If you don't have a Stream Analytics project, follow this guide to create one using Visual Studio Code or export one from Azure.

This article explains how to use the npm package with any CI/CD system. To set up a pipeline with Azure DevOps, see Use Azure DevOps to create a CI/CD pipeline for a Stream Analytics job.

Install

npm install -g azure-streamanalytics-cicd

Build project

[!NOTE] We highly recommend that you use the --v2 option for the updated ARM template schema. The updated schema has fewer parameters yet retains the same functionality as the previous version.

The old ARM template will be deprecated in the future. From now on, only templates that were created via build --v2 will receive updates or bug fixes.

# Generate ARM template
azure-streamanalytics-cicd build --v2 --project <projectFullPath> [--outputPath <outputPath>] 
# Generate Bicep template
azure-streamanalytics-cicd build --v2 --project <projectFullPath> [--outputPath <outputPath>] --type Bicep

The build command does a keyword syntax check and generates Azure Resource Manager (ARM) templates.

| Argument | Description | |---|---| | --project | Specify the asaproj.json file using absolute or relative path. | | --outputPath | Specify the output folder for storing ARM Templates using absolute or relative path. If outputPath is not specified, the templates will be placed in the current directory. | | --type | The template type for Azure resource deployment. Valid values are 'ARM' (for ARM templates) and 'Bicep' (for Bicep templates). Note that Bicep templates are only available with --v2 option and .NET 6.0 environments. If you do not specify a template type, the default is 'ARM'. |

Example:

# Go to the project directory
cd <path-to-the-project>

# Build project to generate ARM template
azure-streamanalytics-cicd build --v2 --project ./asaproj.json --outputPath ./Deploy

# Build project to generate Bicep template
azure-streamanalytics-cicd build --v2 --project ./asaproj.json --outputPath ./Deploy --type Bicep

If the project is built successfully, you see 2 files created under the output folder:

• ARM template
  • ARM template file: [ProjectName].JobTemplate.json
  • ARM parameter file: [ProjectName].JobTemplate.parameters.json
• Bicep
  • Bicep template file: [ProjectName].JobTemplate.bicep
  • Bicep parameter file: [ProjectName].JobTemplate.bicep.parameters.json

The default values for parameters.json file come from your project settings. If you want to deploy to another environment, replace the values accordingly.

The default values for all credentials are null. You are required to set the values before you deploy to Azure.

"Input_EntryStream_sharedAccessPolicyKey": {
  "value": null
}

To use Managed Identity for Azure Data Lake Store Gen1 as output sink, you need to provide Access to the service principal using PowerShell before deploying to Azure. Learn more about how to deploy ADLS Gen1 with Managed Identity with Resource Manager template.

Configure autoscale settings

azure-streamanalytics-cicd autoscale --capacity <maximumSU> --metrics <metricNames> --targetJobName <jobName> [--outputPath <outputPath>]

You can configure autoscale settings for your Stream Analytics job to dynamically adjusts streaming units (SUs) based on your rule definitions.

The CI/CD npm package allows you to specify the maximum SUs and configure a set of rules for autoscaling your jobs. Then it determines whether to add SUs (to handle increases in load) or reduce the number of SUs (when computing resources are sitting idle).

Here're the arguments supported for autoscale command:

| Argument key | Value | Example| |----------------|--------|--------| |--capacity| Maximum SUs (1, 3, 6, or multiples of 6 up to 396)|12| |--metrics | Metric names that defines what the rules monitor, separated by whitespace | ProcessCPUUsagePercentage ResourceUtilization| |--targetJobName| Project name | ClickStream-Filter| |--targetJobResourceId| Job resource ID | - | |--outputPath| Output path for ARM templates | ./Deploy|

[!NOTE] You could either specify the target job name or the target job resource ID. If you specify the target job name, please make sure that you deploy the autoscale settings to the same subscription and resource group as the target job.

Example:

azure-streamanalytics-cicd autoscale --capacity 12 --metrics ProcessCPUUsagePercentage ResourceUtilization --targetJobName ClickStream-Filter --outputPath ./Deploy

If the autoscale setting is configured successfully, you see 2 JSON files created under the output folder:

• ARM template file: [ProjectName].AutoscaleSettingTemplate.json
• ARM parameter file: [ProjectName].AutoscaleSettingTemplate.parameters.json

To learn more about autoscale settings, see Configure autoscale settings for a Stream Analytics job by using the CI/CD tool.

Run locally

If your project has specified local input files, you can run a Stream Analytics script locally by using the localrun command.

azure-streamanalytics-cicd localrun --project <projectFullPath> [--outputPath <outputPath>] [--customCodeZipFilePath <zipFilePath>]

| Argument | Description | |---|---| | --project | Specify the asaproj.json file using absolute or relative path. | | --outputPath | Specify the output folder for storing ARM Templates using absolute or relative path. If outputPath is not specified, the templates will be placed in the current directory. | | --customCodeZipFilePath | The path of the zip file for C# custom code, such as a UDF or deserializer, if they are used. Package the DLLs into a zip file and specify this path. |

Example:

# Go to the project directory
cd <path-to-the-project>

# Run project locally
azure-streamanalytics-cicd localrun --project ./asaproj.json"

[!NOTE] JavaScript UDF and UDA only work on Windows.

Add automated test

You can use the CI/CD npm package to configure and run automated tests for your Stream Analytics project.

Add a test case

azure-streamanalytics-cicd addtestcase --project <projectFullPath> [-testConfigPath <testConfigFileFullPath>]

You can find the test cases in the test configuration file.

| Argument | Description | |---|---| | --project | Specify the asaproj.json file using absolute or relative path. | | --testConfigPath | The path of the test configuration file. If it is not specified, the file will be searched in \test under the current directory of the asaproj.json file, with default file name testConfig.json. A new file will be created if not existed. |

Example:

# Go to the project directory
cd <path-to-the-project>

# Add a test case
azure-streamanalytics-cicd addtestcase --project ./asaproj.json

If the test configuration file is empty, the following content is added to the file. Otherwise, a test case is added to a TestCases array. Necessary input configurations are automatically filled according to the input configuration file. FilePath of each input and expected output must be specified before running the test. You can modify this configuration manually.

If you want the test validation to ignore a certain output, set the Required field of that expected output to false.

{
  "Script": [Absolute path of your script],
  "TestCases": [
    {
      "Name": "Case 1",
      "Inputs": [
        {
          "InputAlias": [Input alias string],
          "Type": "Data Stream",
          "Format": "JSON",
          "FilePath": [Required],
          "ScriptType": "InputMock"
        }
      ],
      "ExpectedOutputs": [
        {
          "OutputAlias": [Output alias string],
          "FilePath": [Required],
          "IgnoreFields": [Fields to ignore for test validation, e.g., ["col1", "col2"]],
          "Required": true
        }
      ]
    }
  ]
}

Run unit test

You can use the following command to run multiple test cases for your project. A summary of test results is generated in the output folder. The process exits with code 0 for all tests passed; -1 for exception occurred; -2 for tests failed.

azure-streamanalytics-cicd test --project <projectFullPath> [--testConfigPath <testConfigFileFullPath>] [--outputPath <outputPath>] [--customCodeZipFilePath <zipFilePath>]

| Argument | Description | |---|---| | --project | The path of the asaproj.json file. | | --testConfigPath | The path to the test configuration file. If it is not specified, the file will be searched in \test under the current directory of the asaproj.json file, with default file name testConfig.json. | --outputPath | The path of the test result output folder. If it is not specified, the output result files will be placed in the current directory. | | --customCodeZipFilePath | The path of the zip file for custom code such as a UDF or deserializer, if they are used. You need to package the DLLs to zip file and specify the path. |

If test cases are executed, you can find a testResultSummary.json file generated in the output folder.

{
  "Total": (integer) total_number_of_test_cases,
  "Passed": (integer) number_of_passed_test_cases,
  "Failed": (integer) number_of_failed_test_cases,
  "Script": (string) absolute_path_to_asaql_file,
  "Results": [ (array) detailed_results_of_test_cases
    {
      "Name": (string) name_of_test_case,
      "Status": (integer) 0(passed)_or_1(failed),
      "Time": (string) time_span_of_running_test_case,
      "OutputMatched": [ (array) records_of_actual_outputs_equal_to_expected_outputs
        {
          "OutputAlias": (string) output_alias,
          "ExpectedOutput": (string) path_to_the_expected_output_file,
          "Output": (string) path_to_the_actual_output_file
        }
      ],
      "OutputNotEqual": [ (array) records_of_actual_outputs_not_equal_to_expected_outputs
        {
          "OutputAlias": (string) output_alias,
          "ExpectedOutput": (string) path_to_the_expected_output_file,
          "Output": (string) path_to_the_actual_output_file
        }
      ],
      "OutputMissing": [ (array) records_of_actual_outputs_missing
        {
          "OutputAlias": (string) output_alias,
          "ExpectedOutput": (string) path_to_the_expected_output_file,
          "Output": ""
        }
      ],
      "OutputUnexpected": [ (array) records_of_actual_outputs_unexpected
        {
          "OutputAlias": (string) output_alias,
          "ExpectedOutput": "",
          "Output": (string) path_to_the_actual_output_file
        }
      ],
      "OutputUnrequired": [ (array) records_of_actual_outputs_unrequired_to_be_checked
        {
          "OutputAlias": (string) output_alias,
          "ExpectedOutput": (string) path_to_the_expected_output_file,
          "Output": (string) path_to_the_actual_output_file
        }
      ]
    }
  ],
  "Time": (string) time_span_of_running_all_test_cases,
}

[!NOTE] If the query results contain float values, you might experience slight differences in the produced values leading to a probably failed test. This is based on the different .Net frameworks powering the Visual Studio or Visual Studio engine and the test processing engine. If you want to make sure that the tests run successfully, you will have to decrease the precision of your produced values or align the results to be compared manually to the generated test results.

Deploy to Azure

To deploy your Stream Analytics project using ARM templates, check out these docs:

• Deploy with a Resource Manager template file and Azure PowerShell.
• Use an object as a parameter in a Resource Manager template.

Next steps

• Continuous integration and Continuous deployment for Azure Stream Analytics
• Set up CI/CD pipeline for Stream Analytics job using Azure Pipelines