DSE Terraform Modules

DSE Terraform modules is designed to be used in conjunction with dse-azure-infrastructure. This repo contains the following modules:

• azurerm-backend
  Helper module to bootstrap Azure storage account to store Terraform state files.
• ds-machine-learning-workspace
  Data science ML module to help bootstrap Azure asset to kickstart a data science project.
• azure-app-service Generic module for create an azure app service and app service plan adhering to Tesco standards.
• azure-container-registry generic module for creating azure container registry.
• azure-data-factory generic module for creating azure data factory.
• azure-key-vault Generic module for creating azure key vault
• azure-kubernetes Generic module for creating kubernetes to Tesco standards.
• azure-machine-learning-networking Module for configuring private links and subnets for Azure Machine Learning workspaces.
• azure-postgresql-server Generic module for created azure postgresql server.
• azure-synapse-workspace Generic module for created azure synapse workspace
• common common references such as tesco ip address ranges

Table of Contents:

• Pre-requisites
• How do you use these modules?
• How do you change a module?
  • Local changes
  • Releasing a new version
• Contributing to Terraform Modules
  • Software Release Cycle
  • Commit Message Guideline
  • Type
• GitHub Actions Pipelines
• TODO

Pre-requisites

Assuming you are using Apple MacOS use homebrew to install all the required packages for development.

# Install Terraform and Terragrunt cli
brew install [email protected] terragrunt

# Packages to run pre-commit checks
brew install node pre-commit coreutils bash  \
             tfsec terraform-docs shellcheck \
             && npm install -g markdown-link-check

# Install older version tflint 0.34.1
# Tflint 0.35.0 breaks API compatibility with azurerm plugin
# brew unlink tflint (if you have another version installed)
brew install ./tflint.rb

Next, install pre-commit packages.

pre-commit install

That's it. Now every time you commit a code change, the hooks: in the hooks: config will execute. If you'd like to format all of your code at once, you can run:

pre-commit run -a

How do you use these modules?

To use a module, create a terragrunt.hcl file that specifies the module you want to use as well as values for the input variables of that module:

# Use Terragrunt to download the module code
terraform {
  source = "git::[email protected]:BUS-DataScience/dse-terraform-modules.git//path/to/module?ref=v0.1.0"
}

# Fill in the variables for that module
inputs = {
  foo = "bar"
  baz = 3
}

Note: the double slash (//) in the source URL is intentional and required. It's part of Terraform's Git syntax for module sources.

You then run Terragrunt, and it will download the source code specified in the source URL into a temporary folder, copy your terragrunt.hcl file into that folder, and run your Terraform command in that folder.

How do you change a module?

Local changes

To test out changes to a module locally:

1. git clone this repo.
2. Update the code as necessary in a new branch.
3. Go into the directory where you have the terragrunt.hcl file in dse-azure-infrastructure that uses a module from this repo (use dev environment!).
4. Run terragrunt plan --terragrunt-source <LOCAL_PATH>, where LOCAL_PATH is the path to your local checkout of the module code.

Using the --terragrunt-source parameter (or TERRAGRUNT_SOURCE environment variable) allows you to do rapid, iterative, make-a-change-and-rerun development.

Releasing a new version

When you're done testing the changes locally, here is how you release a new version:

1. Update the code as required in a new branch.
2. :exclamation: Commit your changes to Git: git commit -m "feat: <COMMIT_MESSAGE>.
   Please review commit message guidelines before writing your commit message.
3. Raise a PR to either alpha or beta or main branch.
4. Once the PR have been approved and merged, the latest update will be tagged and released by semantic-release as defined in our release pipeline.

Contributing to Terraform Modules

Software Release Cycle

• Alpha (alpha): Software may not contain all of the features that are planned for the final version and may contain known or unknown bugs.
• Beta (beta): Software is feature complete but likely to contain a number of known or unknown bugs.
• Stable (main): Software which has passed all verification / tests. The remaining bugs are considered as acceptable.

Commit Message Guideline

We have adopted conventional commits specification for our commit messages to automate our releases. This convention dovetails with SemVer by describing the features, fixes, and breaking changes made in commit messages.

The commit message should be structured as follows:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Type

Must be one of the following:

• build: Changes that affect the build system or external dependencies
• ci: Changes to our CI configuration files and scripts
• docs: Documentation only changes
• feat: A new feature
• fix: A bug fix
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
• test: Adding missing tests or correcting existing tests
• revert: Reverts a previous commit
• chore: Housekeeping tasks that do not cause code changes

For more information about conventional commits please review the official documentation.

GitHub Actions Pipelines

GitHub Actions Pipelines are stored in the .github folder. This repo contains reusbale workflow definitions for Plan & Apply and Pull Request workflows. There are also pipelines within this repo for checking the modules when they are created.

• Pull Request
  This pipeline is triggered when new PR is submitted. When triggered it runs static analysis to check the format of Terraform code.
• Create Release
  This pipeline will be triggered automatically when the code is merged. This uses semantic release to create a new github release see Releases

We use centrally managed GitHub actions worker pool 'enterprise-linux-runners' for both the above mentioned pipelines and also the pipelines in the individual infrastructure repos. Central DevOps Documentation DSE GitHub Actions Documentation

When running on these runners we use docker images with terraform/terragrunt preinstaled which is created in the dse-runner-images repo. These images are tagged with the terraform version they are for.

Currently we are using a shared SSH key for connecting to GitHub this is for pulling the terraform modules. This will be changed to use svc-ds-dev in the future and also we are looking into using GITHUB_TOKEN for this.

TODO

• Setup robust Terraform module testing framework using Terratest. - WIP
• Use GITHUB_TOKEN for pulling terraform modules (removing need for SSH key)