Let’s say you have created a very simple Azure DevOps Pipeline with only STEPS and Tasks. Now you want to add more sophistication.
A pipeline Job helps further organize a set of steps into a logical grouping for easier management. And also provide technical capabilities such as
- Using a specific agent pool such a Linux or Windows based agent in either a MS hosted or self hosted agent in your virtual machine. Also a specific container in an agent pool.
- The ability to run jobs in parrallel,
- The ability to set dependencies, conditions, timeouts at the job level
- The ability to set a set of variables applicable to a job
Reference:https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Specify jobs in your pipeline
I have a pipeline that deploys an Azure resource using Terraform as follows. Note that there is no Job defined in the YAML, but Azure DevOps runs the steps in one single default Job. Some YAML parts omitted for brevity.
trigger:
- main
pool:
vmImage: ubuntu-latest
# Define variables
variables:
- name: environment
value: dev
- name: location
value: canadacentral
<ommitted the rest of the variables>
# Part of the Starter YAML pipeline template and is good example for debugging.
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
# Install Terraform
# Since using MS hosted agent, need to install the terraform and indicate the version.
- task: TerraformInstaller@0
displayName: install terraform
inputs:
terraformVersion: latest
# Configure Azure Provider
# This is optional but just for debugging purposes and test the AzureCLI task
- task: AzureCLI@2
condition: false
inputs:
azureSubscription: $(serviceConnectionName)
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
az account show
# Need to create a resource group for the solution to be deployed. At the minimum need a resource group that is containing the essential storage account for the terraform statefile.
- task: AzureCLI@2
displayName: 'Create resource group $(resource_group_name)'
condition: true
inputs:
azureSubscription: $(serviceConnectionName)
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az group create -n $(resource_group_name) -l $(location)
# A storage account is needed to store the terraform statefile. A Terraform state file is a # file that Terraform uses to store the state of the resources that it manages.
# The state file keeps track of the current state of the infrastructure, and Terraform uses this
# information to determine the changes that need to be made to the infrastructure to bring it to the desired state.
# By placing it in a storage account, promotes collaboration so that other devops engineers can update and run the terraform code.
- task: AzureCLI@2
displayName: 'Create storage account $(storage_account_name) for terraform state files'
condition: true
inputs:
azureSubscription: $(serviceConnectionName)
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az storage account create -n $(storage_account_name) -g $(resource_group_name) -l $(location) --sku Standard_LRS
az storage container create --name terraform-state --account-name $(storage_account_name)
# Initialize Terraform
# This initializes ensuring that Terraform has the necessary information to manage the # infrastructure, including the plugins for the providers that you are using, the state of
# your resources, and the backend to store the state file.
- task: TerraformTaskV2@2
displayName: 'Terraform init'
inputs:
command: 'init'
provider: 'azurerm'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
backendServiceArm: '$(serviceConnectionName)'
backendAzureRmResourceGroupName: '$(resource_group_name)'
backendAzureRmResourceGroupLocation: '$(location)'
backendAzureRmStorageAccountName: '$(storage_account_name)'
backendAzureRmContainerName: 'terraform-state'
backendAzureRmKey: 'terraform.tfstate'
commandOptions: '-lock=false'
# The terraform plan command takes your Terraform configuration as input and compares the desired state specified in the configuration to the current state stored in
# the Terraform state file. It then generates an execution plan, which is a summary of
# the changes that Terraform will make to your infrastructure in order to bring it in line # with the desired state specified in the configuration.
- task: TerraformTaskV3@0
displayName: 'Terraform plan'
inputs:
command: 'plan'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
environmentServiceNameAzureRM: '$(serviceConnectionName)' # Service connection or to override the subscription id defined in a Subscription scoped service connection
commandOptions: '-var "environment=$(environment)" -var "resource_group_name=$(resource_group_name)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name)" -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" -input=false'
# The -input=false option indicates that Terraform should not attempt to prompt for input, and instead expect all necessary values to be provided by either configuration files or the command line
# Provisions the specified resources in your infrastructure, updates existing resources
# as necessary, and deletes any resources that are no longer specified in your Terraform # configuration.
- task: TerraformTaskV3@0
displayName: 'Terraform apply'
inputs:
command: 'apply'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
environmentServiceNameAzureRM: '$(serviceConnectionName)' # Service connection or to override the subscription id defined in a Subscription scoped service connection
commandOptions: '-var "environment=$(environment)" -var "resource_group_name=$(resource_group_name)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name)" -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" -input=false'
# Delete the azure resources under terraform management. This has a conditional flag to manually enable when need.
- task: TerraformTaskV3@0
displayName: 'Terraform destroy'
condition: false # disable destroying
inputs:
command: 'destroy'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
environmentServiceNameAzureRM: '$(serviceConnectionName)' # Service connection or to override the subscription id defined in a Subscription scoped service connection
commandOptions: '-var "environment=$(environment)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name) -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" "'
- script: |
echo Finished AZDO pipeline demo execution
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
Running the pipeline shows only one job by default yet runs the steps and tasks within it.
The following is simply re-organize the steps into 3 jobs. Note that you need a steps: line nested under each job.
trigger:
- main
pool:
vmImage: ubuntu-latest
# Define variables
variables:
- name: environment
value: dev
- name: location
value: canadacentral
- name: subscriptionId
<omitted>
jobs:
- job: BashScript
steps:
- script: echo Hello, world to my pipeline!
displayName: 'Bash script'
- job: Setup
# Install Terraform
# Since using MS hosted agent, need to install the terraform and indicate the version.
steps:
- task: TerraformInstaller@0
displayName: install terraform
inputs:
terraformVersion: latest
# Configure Azure Provider
- task: AzureCLI@2
condition: false
inputs:
azureSubscription: $(serviceConnectionName)
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
az account show
- task: AzureCLI@2
displayName: 'Create resource group $(resource_group_name)'
condition: true
inputs:
azureSubscription: $(serviceConnectionName)
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az group create -n $(resource_group_name) -l $(location)
- task: AzureCLI@2
displayName: 'Create storage account $(storage_account_name) for terraform state files'
condition: true
inputs:
azureSubscription: $(serviceConnectionName)
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az storage account create -n $(storage_account_name) -g $(resource_group_name) -l $(location) --sku Standard_LRS
az storage container create --name terraform-state --account-name $(storage_account_name)
- job: TerraformTasks
steps:
- task: TerraformTaskV2@2
displayName: 'Terraform init'
inputs:
command: 'init'
provider: 'azurerm'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
backendServiceArm: '$(serviceConnectionName)'
backendAzureRmResourceGroupName: '$(resource_group_name)'
backendAzureRmResourceGroupLocation: '$(location)'
backendAzureRmStorageAccountName: '$(storage_account_name)'
backendAzureRmContainerName: 'terraform-state'
backendAzureRmKey: 'terraform.tfstate'
commandOptions: '-lock=false'
- task: TerraformTaskV3@0
displayName: 'Terraform plan'
inputs:
command: 'plan'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
environmentServiceNameAzureRM: '$(serviceConnectionName)' # Service connection or to override the subscription id defined in a Subscription scoped service connection
commandOptions: '-var "environment=$(environment)" -var "resource_group_name=$(resource_group_name)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name)" -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" -input=false'
- task: TerraformTaskV3@0
displayName: 'Terraform apply'
inputs:
command: 'apply'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
environmentServiceNameAzureRM: '$(serviceConnectionName)' # Service connection or to override the subscription id defined in a Subscription scoped service connection
commandOptions: '-var "environment=$(environment)" -var "resource_group_name=$(resource_group_name)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name)" -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" -input=false'
- task: TerraformTaskV3@0
displayName: 'Terraform destroy'
condition: false # disable destroying
inputs:
command: 'destroy'
workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
environmentServiceNameAzureRM: '$(serviceConnectionName)' # Service connection or to override the subscription id defined in a Subscription scoped service connection
commandOptions: '-var "environment=$(environment)" -var "location=$(location)" -var "functionapp_storage_account_name=$(functionapp_storage_account_name) -var "azurerm_windows_function_app_name=$(azurerm_windows_function_app_name)" "'
- script: |
echo Finished AZDO pipeline demo execution
echo See https://aka.ms/yaml
displayName: 'The End'
Hope this provided the next step overview and understanding in starting out to apply multiple jobs and continue to build on it with added capabilities supported at the job level. The next advancement in developing a robust pipeline solution is to consider Pipeline Stages.
Roy Kim on Azure and Microsoft 365 