Azure DevOps - Pipeline as code

CICD pipeline as code using Azure DevOps REST APIs with Postman

Hooker Valley, New Zealand by author

Why

To provision a source code repository and CICD pipelines in Azure DevOps for a typical Spring Boot based micro service requires a lot of mouse clicks. It is not only time-consuming but also error-prone.

Figure 1. linked variable groups make access control of Externalised Configuration a trivial task

Impact of pipeline as code

Prior to the adoption of this approach, a typical provisioning process for a new micro service often took more than half an hour. We also have discrepancies among pipelines for different micro services that are authored by different engineers. Confusion hindered productivity and human errors in repository/pipeline configuration frustrated engineers in many occasions.

How

At the heart of this approach, there are Azure DevOps REST API endpoints and a number of JSON payloads.

Steps

  1. Create a repo with two branches: master and dev
Figure 2. a master branch that follows common CICD practise
  • pull requests must be reviewed by other engineer(s) before merge
  • comments on a pull request must be resolved before merge
1. Create a repoEndpoint:
https://dev.azure.com/{{collection}}/{{project}}/_apis/git/repositories?api-version=5.0
Payload:{
"name": "{{service_name}}",
"defaultBranch": "refs/heads/master"
}
Post-response Postman Test Script://To save crucial response data to postman variables pm.globals.set("repo_id", pm.response.json().id);
pm.globals.set("project_id", pm.response.json().project.id);
2. Initiate master branchEndpoint:https://dev.azure.com/{{collection}}/{{project}}/_apis/git/repositories/{{repo_id}}/pushes?api-version=5.0Payload:{
"name": "{{service_name}}",
"defaultBranch": "refs/heads/master"
}
Post-response Postman Test Script: pm.globals.set("master_branch", pm.response.json().refUpdates[0].newObjectId);3. Master branch policy - PR must be reviewed and approvedEndpoint:https://voltbank.visualstudio.com/{{project_id}}/_apis/policy/Configurations?api-version=5.0Payload:{
"type": {
"id": "fa4e907d-c16b-4a4c-9dfa-4906e5d171dd"
},
"isDeleted": false,
"isBlocking": true,
"isEnabled": true,
"settings": {
"minimumApproverCount": 1,
"creatorVoteCounts": false,
"allowDownvotes": false,
"resetOnSourcePush": false,
"scope": [
{
"refName": "refs/heads/master",
"matchKind": "Exact",
"repositoryId": "{{repo_id}}"
}
]
}
}
4. Master branch policy - comments must be resolved before mergeEndpoint:https://voltbank.visualstudio.com/{{project_id}}/_apis/policy/Configurations?api-version=5.0Payload:
{
"isBlocking": true,
"isDeleted": false,
"isEnabled": true,
"revision": 1,
"settings": {
"scope": [
{
"matchKind": "Exact",
"refName": "refs/heads/master",
"repositoryId": "{{repo_id}}"
}
]
},
"type": {
"id": "c6a1889d-b943-4856-b76f-9e46bb6b0df2"
}
}
5. Create dev branchEndpoint:https://dev.azure.com/{{collection}}/{{project}}/_apis/git/repositories/{{repo_id}}/refs?api-version=5.0Payload:[
{
"name": "refs/heads/dev",
"newObjectId": "{{master_branch}}",
"oldObjectId": "0000000000000000000000000000000000000000"
}
]
Figure 3. build pipeline
  • It will be triggered by a merge to master branch of newly created repository
Endpoint:
https://dev.azure.com/{{collection}}/{{project}}/_apis/build/definitions?api-version=5.0
Payload:{
"triggers": [
{
"branchFilters": [
"+refs/heads/master"
],
"maxConcurrentBuildsPerBranch": 1,
"pollingInterval": 0,
"triggerType": 2
}
],
"variables": {{build_variables}},
"retentionRules": [
{
"branches": [
"+refs/heads/*"
],
"daysToKeep": 10,
"minimumToKeep": 1,
"deleteBuildRecord": true,
"deleteTestResults": true
}
],
"buildNumberFormat": "$(date:yyyyMMdd)$(rev:.r)",
"jobAuthorizationScope": 1,
"jobTimeoutInMinutes": 60,
"jobCancelTimeoutInMinutes": 5,
"process": {
"phases": [
{
"steps": [
{
"enabled": true,
"continueOnError": false,
"displayName": "Package & SpotBugs",
"condition": "succeeded()",
"task": {
"id": "ac4ee482-65da-4485-a532-7b085873e532",
"versionSpec": "3.*",
"definitionType": "task"
},
"inputs": {
"mavenPOMFile": "$(Parameters.mavenPOMFile)",
"goals": "package spotbugs:spotbugs",
"options": "-Dhttp.keepAlive=false",
"publishJUnitResults": "true",
"testResultsFiles": "target/surefire-reports/TEST-*.xml",
"codeCoverageTool": "Cobertura",
"failIfCoverageEmpty": "false",
"javaHomeSelection": "JDKVersion",
"jdkVersion": "default",
"jdkArchitecture": "x64",
"mavenVersionSelection": "Default",
"mavenSetM2Home": "false",
"mavenOpts": "-Xmx3072m",
"mavenFeedAuthenticate": "true",
"skipEffectivePom": "false",
"sqAnalysisEnabled": "false",
"sqMavenPluginVersionChoice": "latest",
"checkstyleAnalysisEnabled": "false",
"pmdAnalysisEnabled": "false",
"findbugsAnalysisEnabled": "false"
}
},
{
"enabled": true,
"continueOnError": false,
"displayName": "Build an image",
"condition": "succeeded()",
"task": {
"id": "e28912f1-0114-4464-802a-a3a35437fd16",
"versionSpec": "0.*",
"definitionType": "task"
},
"inputs": {
"containerregistrytype": "Azure Container Registry",
"dockerRegistryEndpoint": "",
"azureSubscriptionEndpoint": "{{azure_service_connection_id}}",
"azureContainerRegistry": "{\"loginServer\":\"{{docker_name_space}}\", \"id\" : \"/subscriptions/{{docker_subscription}}/resourceGroups/{{docker_resource_group}}/providers/Microsoft.ContainerRegistry/registries/{{docker_registry}}\"}",
"action": "Build an image",
"dockerFile": "**/Dockerfile",
"defaultContext": "true",
"imageName": "{{docker_name_space}}/{{docker_repository}}:$(Build.BuildNumber)",
"qualifyImageName": "true",
"includeSourceTags": "false",
"includeLatestTag": "false",
"detached": "true",
"restartPolicy": "no",
"enforceDockerNamingConvention": "true",
"cwd": "$(System.DefaultWorkingDirectory)"
}
},
{
"enabled": true,
"continueOnError": false,
"displayName": "Push an image",
"condition": "succeeded()",
"task": {
"id": "e28912f1-0114-4464-802a-a3a35437fd16",
"versionSpec": "0.*",
"definitionType": "task"
},
"inputs": {
"containerregistrytype": "Azure Container Registry",
"dockerRegistryEndpoint": "",
"azureSubscriptionEndpoint": "{{azure_service_connection_id}}",
"azureContainerRegistry": "{\"loginServer\":\"{{docker_name_space}}\", \"id\" : \"/subscriptions/{{docker_subscription}}/resourceGroups/{{docker_resource_group}}/providers/Microsoft.ContainerRegistry/registries/{{docker_registry}}\"}",
"action": "Push an image",
"dockerFile": "**/Dockerfile",
"defaultContext": "true",
"imageName": "{{docker_name_space}}/{{docker_repository}}:$(Build.BuildNumber)",
"qualifyImageName": "true",
"detached": "true",
"restartPolicy": "no",
"enforceDockerNamingConvention": "true",
"cwd": "$(System.DefaultWorkingDirectory)"
}
}
],
"name": "Build",
"refName": "Phase_1",
"condition": "succeeded()",
"target": {
"executionOptions": {
"type": 0
},
"allowScriptsAuthAccessOption": false,
"type": 1
},
"jobAuthorizationScope": 1,
"jobCancelTimeoutInMinutes": 1
}
],
"type": 1
},
"repository": {
"properties": {
"labelSources": "6",
"labelSourcesFormat": "$(build.buildNumber)",
"reportBuildStatus": "true",
"fetchDepth": "0",
"cleanOptions": "3",
"gitLfsSupport": "false",
"skipSyncSource": "false",
"checkoutNestedSubmodules": "false"
},
"id": "{{repo_id}}",
"type": "TfsGit",
"name": "{{service_name}}",
"defaultBranch": "refs/heads/master",
"clean": "true",
"checkoutSubmodules": false
},
"processParameters": {
"inputs": [
{
"name": "mavenPOMFile",
"label": "Maven POM file",
"defaultValue": "pom.xml",
"required": true,
"type": "filePath",
"helpMarkDown": ""
}
]
},
"queue": {
"id": 155,
"name": "Hosted Ubuntu 1604",
"pool": {
"id": 6,
"name": "Hosted Ubuntu 1604",
"isHosted": true
}
},
"name": "{{service_name}}-master",
"path": "\\master\\",
"type": 2,
"project": {
"id": "{{project_id}}"
}
}
Post-response Postman Test Script:
pm.globals.set("build_pipeline_id", pm.response.json().id);
  • A deployment to dev environment is triggered whenever a new Docker image is published by newly created build pipeline
Figure 4. this pipeline kicks off a new release by automatically deploying to dev environment
Figure 5. deployments to test and staging environment are guarded by Approvers group
  • App Services retrieve environment specific values using the same group of keys from environment specific Azure Key Vaults. Access to Azure Key Vaults is controlled by the scopes of “linked variable groups”
Figure 1 (again) linked variable groups from Azure Key Vaults
Endpoint:https://dev.azure.com/{{collection}}/{{project}}/_apis/build/definitions?api-version=5.0Payload:
{
"environments": [
{
"name": "dev",
"rank": 1,
"owner": {
"id": "{{author_id}}"
},
"variableGroups": {{linked_variable_groups_dev}},
"preDeployApprovals": {
"approvals": [
{
"rank": 1,
"isAutomated": true
}
],
"approvalOptions": {
"executionOrder": 1
}
},
"postDeployApprovals": {
"approvals": [
{
"rank": 1,
"isAutomated": true
}
],
"approvalOptions": {
"executionOrder": 2
}
},
"deployPhases": [
{
"deploymentInput": {
"skipArtifactsDownload": false,
"queueId": 155,
"jobCancelTimeoutInMinutes": 1,
"condition": "succeeded()"
},
"rank": 1,
"phaseType": 1,
"name": "Agent phase",
"workflowTasks": [
{
"taskId": "497d490f-eea7-4f2b-ab94-48d9c1acdcb1",
"version": "4.*",
"name": "Deploy Azure App Service Environment",
"enabled": true,
"definitionType": "task",
"condition": "succeeded()",
"inputs": {
"ConnectionType": "AzureRM",
"ConnectedServiceName": "{{azure_service_connection_id}}",
"PublishProfilePath": "$(System.DefaultWorkingDirectory)/**/*.pubxml",
"PublishProfilePassword": "",
"WebAppKind": "webAppContainer",
"WebAppName": "{{web_app_name_dev}}",
"DeployToSlotOrASEFlag": "false",
"ResourceGroupName": "{{rss_group_name_dev}}",
"DockerNamespace": "{{docker_name_space}}",
"DockerRepository": "{{docker_repository}}",
"DockerImageTag": "$(BUILD.BUILDID)",
"Package": "$(System.DefaultWorkingDirectory)/**/*.zip",
"AppSettings": "{{app_settings_dev}}",
"UseWebDeploy": "false",
"DeploymentType": "webDeploy"
}
}
]
}
],
"conditions": [
{
"name": "ReleaseStarted",
"conditionType": 1,
"value": ""
}
],
"retentionPolicy": {
"daysToKeep": 30,
"releasesToKeep": 3,
"retainBuild": true
},
"executionPolicy": {
"concurrencyCount": 1,
"queueDepthCount": 0
}
},
{
"name": "test",
"rank": 2,
"owner": {
"id": "{{author_id}}"
},
"variableGroups": {{linked_variable_groups_test}},
"preDeployApprovals": {
"approvals": [
{
"rank": 1,
"isAutomated": false,
"approver": {
"id": "{{approvers_group_id}}",
"isContainer": true
}
}
],
"approvalOptions": {
"requiredApproverCount": 1,
"releaseCreatorCanBeApprover": true,
"executionOrder": 1
}
},
"postDeployApprovals": {
"approvals": [
{
"rank": 1,
"isAutomated": true
}
],
"approvalOptions": {
"executionOrder": 2
}
},
"deployPhases": [
{
"deploymentInput": {
"skipArtifactsDownload": false,
"queueId": 155,
"jobCancelTimeoutInMinutes": 1,
"condition": "succeeded()"
},
"rank": 1,
"phaseType": 1,
"name": "Agent phase",
"workflowTasks": [
{
"taskId": "497d490f-eea7-4f2b-ab94-48d9c1acdcb1",
"version": "4.*",
"name": "Deploy Azure App Service Environment",
"enabled": true,
"definitionType": "task",
"condition": "succeeded()",
"inputs": {
"ConnectionType": "AzureRM",
"ConnectedServiceName": "{{azure_service_connection_id}}",
"PublishProfilePath": "$(System.DefaultWorkingDirectory)/**/*.pubxml",
"PublishProfilePassword": "",
"WebAppKind": "webAppContainer",
"WebAppName": "{{web_app_name_test}}",
"DeployToSlotOrASEFlag": "false",
"ResourceGroupName": "{{rss_group_name_test}}",
"DockerNamespace": "{{docker_name_space}}",
"DockerRepository": "{{docker_repository}}",
"DockerImageTag": "$(BUILD.BUILDID)",
"Package": "$(System.DefaultWorkingDirectory)/**/*.zip",
"AppSettings": "{{app_settings_test}}",
"UseWebDeploy": "false",
"DeploymentType": "webDeploy"
}
}
]
}
],
"conditions": [
{
"name": "dev",
"conditionType": 2,
"value": "4"
}
],
"executionPolicy": {
"concurrencyCount": 1,
"queueDepthCount": 1
},
"retentionPolicy": {
"daysToKeep": 30,
"releasesToKeep": 3,
"retainBuild": true
}
},
{
"name": "staging",
"rank": 3,
"owner": {
"id": "{{author_id}}"
},
"variableGroups": {{linked_variable_groups_stg}},
"preDeployApprovals": {
"approvals": [
{
"rank": 1,
"isAutomated": false,
"approver": {
"id": "{{approvers_group_id}}",
"isContainer": true
}
}
],
"approvalOptions": {
"executionOrder": 1
}
},
"postDeployApprovals": {
"approvals": [
{
"rank": 1,
"isAutomated": true
}
],
"approvalOptions": {
"executionOrder": 2
}
},
"deployPhases": [
{
"deploymentInput": {
"queueId": 155,
"jobCancelTimeoutInMinutes": 1,
"condition": "succeeded()"
},
"rank": 1,
"phaseType": 1,
"name": "Agent phase",
"refName": null,
"workflowTasks": [
{
"taskId": "497d490f-eea7-4f2b-ab94-48d9c1acdcb1",
"version": "4.*",
"name": "Deploy Azure App Service Environment",
"enabled": true,
"definitionType": "task",
"condition": "succeeded()",
"inputs": {
"ConnectionType": "AzureRM",
"ConnectedServiceName": "{{azure_service_connection_id}}",
"PublishProfilePath": "$(System.DefaultWorkingDirectory)/**/*.pubxml",
"PublishProfilePassword": "",
"WebAppKind": "webAppContainer",
"WebAppName": "{{web_app_name_stg}}",
"DeployToSlotOrASEFlag": "false",
"ResourceGroupName": "{{rss_group_name_stg}}",
"DockerNamespace": "{{docker_name_space}}",
"DockerRepository": "{{docker_repository}}",
"DockerImageTag": "$(BUILD.BUILDID)",
"Package": "$(System.DefaultWorkingDirectory)/**/*.zip",
"AppSettings": "{{app_settings_stg}}",
"UseWebDeploy": "false",
"DeploymentType": "webDeploy"
}
}
]
}
],
"conditions": [
{
"name": "test",
"conditionType": 2,
"value": "4"
}
],
"executionPolicy": {
"concurrencyCount": 1,
"queueDepthCount": 1
},
"retentionPolicy": {
"daysToKeep": 30,
"releasesToKeep": 3,
"retainBuild": true
}
}
],
"artifacts": [
{
"sourceId": "{{azure_service_connection_id}}:{{docker_name_space}}:{{docker_repository}}",
"type": "AzureContainerRepository",
"alias": "_{{collection}}_{{service_name}}",
"definitionReference": {
"connection": {
"id": "{{azure_service_connection_id}}",
"name": "Azure DevOps Service Principal"
},
"defaultVersionType": {
"id": "latestType",
"name": "Latest"
},
"definition": {
"id": "{{docker_repository}}",
"name": "{{docker_repository}}"
},
"registryurl": {
"id": "{{docker_name_space}}",
"name": "{{docker_registry}}"
},
"resourcegroup": {
"id": "{{docker_resource_group}}",
"name": "{{docker_resource_group}}"
}
},
"isPrimary": true,
"isRetained": false
}
],
"triggers": [
{
"alias": "_{{collection}}_{{service_name}}",
"triggerType": 4
}
],
"releaseNameFormat": "Release-$(rev:r)",
"tags": [],
"properties": {
"DefinitionCreationSource": {
"$type": "System.String",
"$value": "ReleaseClone"
},
"IntegrateJiraWorkItems": {
"$type": "System.String",
"$value": "false"
}
},
"name": "{{service_name}} - release",
"path": "\\Development"
}

Summary

To sum up, the application of pipeline as code for Azure DevOps reduced the provisioning time by 90% for us and has brought peace of mind to engineers. It is pretty incredible.

Figure 6 Postman collection imported from my github repo

Father of two giggly girls; a technical problem solver who focuses on both delivery and growth

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store