Leveraging Azure DevOps across the Enterprise

Published in

Azure Architects

11 min readDec 19, 2018

This is a fairly lengthy post — if you’d prefer this in presentation format I’ve summarised the lessons learned and recommendations on SlideShare.

Overview

Many of the digital and cloud transformation projects I’ve worked on have historically been managed through a motley collection of project plans, spreadsheets and documents.

Just like any project, if you’ve worked within large enterprises you’ll be familiar with how difficult this can be to manage. Common challenges include —

Changing requirements and ad hoc requests
Multiple vendors and ISV’s
Teams distributed across multiple geographic locations
Inability to collaborate effectively due to disparate technologies

I’m seeing Azure platform teams working across a variety of different Azure-related activities. This could be core Azure infrastructure, database and network configuration, migrating IaaS workloads, re-platforming legacy applications and working with developers to provision environments.

An organisation’s “Azure platform” team is often a multi-discipline team that has capabilities across infrastructure, databases, networking and security that all form a capability to deploy and support workloads on the Azure platform

Often these requirements are dynamic and changing in nature. My experience is that, as agile practices start to extend outside developer teams and into the wider business, platform teams can also gain considerable value in adopting those agile and DevOps methodologies.

To help facilitate this I’m a big fan of Azure DevOps (formerly Visual Studio Team Services) to track and plan activities, store ARM templates and scripts, and manage Azure deployments.

Whilst Azure DevOps (AzDO) won’t be the right fit for every organisation if you’re working on an Azure project then it’s definitely worth a look.

In the remainder of this post, we’ll take a look at the different services within AzDO, how you can leverage them for Azure-related configuration activities and some lessons learned.

There are quite a few different services available within AzDO but we’ll cover the most commonly used ones -

AzDO’s Organisational Structure

Before we dig into the different services it’s worth covering the key organisational concepts within AzDO -

Organisation — a “container” for an organisation’s projects
Projects — reside within a single organisation. It’s possible to have many projects within a single organisation
Teams — a project can contain multiple teams to segregate backlogs and track activities

Some companies have multiple organisations and as an end user it’s possible to be a member of multiple teams across multiple projects located in multiple organisations!

If working within an existing AzDO project, work with your admin to create a dedicated team — this will give you your own dashboard, backlog and board.

In the following example, we can see the MyWidgets company has a single AzDO organisation, with multiple projects and teams.

Okay, let’s explore the different services…

Dashboards

Within the Overview section are two features worth taking a look at.

Firstly, dashboards provide the ability to share real-time progress and status across your team or project. Dashboards are fully customisable using built-in widgets.

Examples of platform-related activities you might want to track through the various widgets include -

List the backlog items that are assigned to you
Progress against test plans
Status of your build and release pipelines

One of the powerful features of dashboards is the ability to link widgets to your own custom queries. For example, you could have a widget that displayed the total number of completed tasks (backlog items) over the last 7 days.

Wiki

Also within the Overview section is the project Wiki. The Wiki provides a convenient location for your project team to distribute information and share knowledge.

The Wiki supports markdown to format pages. This means you can easily add links, tables and images.

I’m not yet convinced that it’ll replace documents that typically need to go through a review and governance process i.e. high and low-level designs. Although design documents provide a useful reference they often get lost in a document repository. This is where the Wiki can add value by making key information easily accessible.

Examples of information I’ve found useful to capture in the Wiki include -

Azure naming standards
Tagging requirements
Descriptions of build and release pipelines
Descriptions of key Azure resources i.e. backup vaults, log analytics workspaces etc.
Documentation for any key ARM templates within the project’s repositories
Operational information
Release notes
FAQ’s

Boards

Boards enable you to track and plan tasks and activities. If you’re familiar with Agile project methodologies then you’ll feel right at home.

A board is essentially a Kanban board that helps you visualise your team’s backlog work items.

Even though boards were originally intended for app dev teams they can work well for other teams.

Below is an example board that shows the type of Azure related activities a platform team could track -

Key things to note for boards are -

New work items get created on the left and move to the right as they’re completed and marked as “done”
Columns are customisable and new ones can be added
Work items and tasks can be assigned to individuals (they’ll get an email notification when this happens)
Tag people using @name to notify someone, request comments etc.
Create tasks within a work item to group similar activities

There are two common methodologies that your AzDO’s project can use, Scrum or Agile. For work items, Scrum uses Backlog items whilst Agile uses User Stories. I personally prefer Scrum. But if you’re working within a wider project or organisation that dictates the use of Agile that’s fine too.

Sprints

Boards are a good starting point for tracking activities but they’re also a bit ad hoc. Therefore it’s possible your team will want to plan when certain activities get worked on.

Traditionally, waterfall project methodologies are used where there is a sequential, linear approach to completing activities.

Sprints take a different approach by adopting a time-boxed cycle of work; 2–3 weeks per sprint is common and normally less than 4 weeks.

For each sprint, the product owner and team agree on what work items should be moved from the product backlog into the sprint backlog. During that sprint, the team will focus on those agreed items.

This is a common approach for app dev teams but there’s also potential to utilise sprints for other types of teams.

As an example take the following scenario. You’re part of a platform team that is supporting several cloud transformation initiatives within your organisation. You have lots of competing requests for Azure environments, infrastructure configuration etc. Some of these activities may only take a few hours whilst others may take a few days or longer.

By agreeing on which activities and requests your team works on during the next sprint you’ll be able to efficiently prioritise the most important activities.

In the example below we’ve assigned some of our backlog items to Sprint 1 which runs for 10 days (9th — 22nd Dec) -

At the end of the sprint, you’ll complete a sprint review to review what work was completed, what worked well etc. You then enter into sprint planning for the next sprint to agree on what activities will be focused on next.

Adopting this mindset and approach can take a bit of getting used to but after a few sprints, you’ll quickly get a feel for it.

I’d recommend starting with a simple board and if additional structure or planning is needed look at leveraging sprints as needed.

Repos

In my opinion, if there’s just one AzDO service you should leverage its Repos. These provide a shared AzDO-hosted Git repository for all of your infrastructure-as-code artefacts; Bicep/ARM templates, Terraform configuration files and PowerShell scripts.

Of course, being Git-based you have the ability to create branches, commit code and create pull requests into a master branch.

Things to consider include -

You can have multiple repos within an AzDO project — create a dedicated “infrastructure” repo for core Azure services
Use feature branches when updating/creating IaC
Assess whether approvals and reviews are needed when pulling updates into the master branch
Agree on a consistent folder structure
Use Visual Studio Code as your IDE to get tight integration with AzDO

In the example below we’re using a folder structure for our ARM templates that align to the different Azure providers-

Pipelines

Having all of our ARM templates and PowerShell scripts within a repo is a great first step on the Infrastructure as Code journey. However, deploying those templates manually via PowerShell or the Azure Portal provides limited audit’ability and manageability.

Pipelines are a very flexible capability that can manage the deployment of those templates. The two key features we’ll cover are Builds and Releases.

Builds

A build definition automatically builds and validates the code in our repository. As our “code” consists of ARM templates and PowerShell scripts our build can be relatively simple. In the example below our build has two tasks which -

Copies the content of our infrastructure repository
Publishes the content as an “artefact”

You can think of the build artefact as a snapshot of our repo which we’ll reference from our release pipelines.

If you’re just creating a simple build for your ARM templates etc. you shouldn’t need to revisit it that often. Typically, you’ll enable Continuous Integration (CI) on your build. This means that when an ARM template or script is updated within the repository a new build is automatically created that contains your updated code.

Releases

A release pipeline provides a way to selectively deploy the ARM templates or scripts contained within our build artefact.

For example, if we need to deploy an Azure site-to-site VPN into each of our subscriptions (Sandbox, PreProd, Prod) we’d create a release pipeline similar to the one below -

In this example we named our stages according to the different Azure subscriptions we’re deploying into. But if you’re deploying an application environment these might be named Dev/Test/UAT/PreProd/Prod etc.

Within each stage, we have a single Azure Resource Group Deployment task that deploys the S2S VPN ARM template from our build artefact -

Whilst we may only need one general-purpose infrastructure build, you’ll likely end up with multiple release pipelines that reference it. For example, some of the pipelines you create may include -

Deploy an ExpressRoute circuit
Deploy Virtual Networks + subnets
Deploy Storage accounts
Deploy Automation accounts
Deploy Log Analytics workspaces
etc.

Some of these resources you’ll typically only want to deploy once, i.e. an ExpressRoute circuit. Others may be updated semi-regularly, i.e. additional subnets added to a VNET. This workflow is outlined below -

Because the underlying ARM templates are declarative, we simply create a new release from our pipeline, and Azure will work out the configuration updates it needs to apply.

Other things to consider for release pipelines include -

How to structure and name the stages, i.e. per subscription or per environment
Stage approvals; leverage these to ensure that only specific individuals can approve a deployment (especially for production!)
Store secrets and certificates for your pipelines in Key Vault
Use the Wiki to document any particularly complex pipelines

Finally, think carefully about whether you want to enable Continuous Delivery (CD) on a pipeline. It is useful for app dev environments that need to automatically deploy and update resources when new code gets released. But this can potentially be quite dangerous if the pipeline makes significant changes to the underlying Azure infrastructure.

Test Plans

When deploying ARM templates via AzDO there’s commonly two types of testing we want to perform -

Deployment testing, i.e. has the template been successfully deployed?
Post-deployment testing, i.e. are the deployed Azure services operating as we’d expect?

For the first type of testing, the release pipeline will automatically generate errors if the template deployment fails. There are also additional tasks we can integrate such as Pester and the Secure DevOps Kit to perform further validation.

For post-deployment testing, we can leverage AzDO’s test plans. These provide the ability to write and track manual testing activities.

Sticking with the example of the site-to-site VPN, let’s assume our release pipeline has successfully deployed the required Azure services (VPN gateway, connection and local network gateway). We now want to perform a series of manual tests to validate that the connection is operational. These tests might include -

Validate the VPN connection status is “connected” within the Azure Portal
RDP connectivity from on-premise to an Azure VM
ICMP ping test from on-premise to an Azure VM

Within test plans, we write our test cases, i.e. -

When ready to start testing, select run and AzDO’s web-based test runner application fires up and guides us through the test steps -

Key reasons I like Test Plans are -

Test cases can be linked to work items
Bugs can be created from failed tests
The Test Runner interface makes it easy to grab screenshots and screen recordings
It’s all within AzDO — no test spreadsheets or 3rd party tools required!

Overall, Test Plans give us the ability to structure and capture the results of our testing activities.

Summary

This post provides a very high-level overview of the services available within AzDO. But hopefully, it gives you a flavour of what’s possible outside its historical remit of a tool to support application development.

By utilising the services within AzDO the platform team will become -

More efficient, transparent and responsive across its activities
Be more effective when planning work
Enable a single, managed end-to-end approach for Infrastructure as Code
Ensure the proper governance is wrapped around Azure deployments

This post has been written in the context of a single Azure platform team. However, many organisations aren’t structured in this way and have those Azure skills dispersed amongst a more traditional team structure, i.e. infrastructure, network, storage etc. In those scenarios, I believe those teams will gain even greater benefit from AzDO — by working out of a single, common tool, they’ll be able to work more effectively together.

You don’t have to go all-in at once either; utilising Repos and Pipelines is an excellent place to start. Start to explore the other services and find what works for you and your teams.