Innovative Approach to AKS DR Using Azure Pipeline Decorator

Introduction

In the realm of cloud computing, ensuring the high availability and disaster recovery (DR) of applications is crucial. Azure Kubernetes Service (AKS) offers robust solutions for deploying, managing, and scaling containerized applications using Kubernetes. However, the traditional method of implementing DR for AKS involves creating a separate AKS cluster in a different region and deploying applications there. This is typically done by adding a task to your pipelines, which deploys applications to the DR region after your production deployment is complete.

While this approach works well for a few pipelines, it becomes impractical for large-scale environments, such as the one at Bajaj FinServ Health, where there are hundreds of micro-services, each with its own repository and pipeline for production releases. Manually adding another task to every pipeline to handle DR deployment is not feasible.

To address this challenge, we have leveraged the Azure Pipeline Decorator, a powerful tool that allows for injecting custom tasks into any pipeline or release, pre or post any task, as defined by its own task ID. This article explores how we implemented this innovative solution.

Understanding Azure Pipeline Decorator

Azure Pipeline Decorators are scripts that can automatically add tasks to the beginning or end of every pipeline or task. This feature is particularly useful in scenarios where you need to enforce certain steps across multiple pipelines without manually modifying each one. In the context of DR for AKS, we can use decorators to ensure that every deployment to the production AKS cluster is followed by a deployment to the DR AKS cluster.

Benefits of Using Azure Pipeline Decorator

• Centralised Management: Manage the DR deployment process from a single script, eliminating the need to modify individual pipelines.
• Consistency: Ensure that DR deployments are consistently applied across all micro-services.
• Scalability: Easily scale the DR deployment process as the number of micro-services grows.
• Simplicity: Reduce the complexity and overhead associated with managing multiple pipelines.

Implementing the Solution

Creating the Extension Configuration

To create an Azure Pipeline decorator, we start by creating a VSS extension configuration file. Below is an example:

{
    "manifestVersion": 1,
    "id": "AKS-DR-Decorator",
    "name": "AKS-DR-Decorator",
    "version": "1.0.2",
    "publisher": "SidharathBansal",
    "targets": [
        {
            "id": "Microsoft.VisualStudio.Services"
        }
    ],
    "icons": {
        "default": "images/decorator-logo.png"
    },
    "description": "Helm Deploy to AKS in DR Region",
    "categories": [
        "Azure Pipelines"
    ],
    "contributions": [
        {
            "id": "AKS-DR-Decorator",
            "type": "ms.azure-pipelines.pipeline-decorator",
            "targets": [
                "ms.azure-pipelines-agent-job.post-task-tasks",
                "ms.azure-release-pipelines-agent-job.post-task-tasks"
            ],
            "properties": {
                "name": "decorator/AKS-DR-Decorator.yml",
                "template": "decorator/AKS-DR-Decorator.yml",
                "targettask": "068D5909-43E6-48C5-9E01-7C8A94816220"
            }
        }
    ],
    "files": [
        {
            "path": "decorator/AKS-DR-Decorator.yml",
            "addressable": true,
            "contentType": "text/plain"
        }
    ]
}

The above example configuration dictates that the decorator should inject the mentioned YAML file after the “068D5909–43E6–48C5–9E01–7C8A94816220” target task, which is the task ID for the Helm installer task in both Azure Pipelines and Azure Releases.

Example YAML File

In the above extension configuration, we have mentioned a YAML file. That YAML contains all the tasks that we need to inject in all pipelines. Here’s an example YAML file containing the required tasks:

steps:
- task: Kubernetes@1
  displayName: 'Kubernetes DR Login (DevOps Task)'
  inputs:
    connectionType: 'Kubernetes Service Connection'
    kubernetesServiceEndpoint: 'DR AKS'
    command: 'login'

- task: CmdLine@2
  continueOnError: true
  displayName: 'DR Deploy (DevOps Task)'
  inputs:
    script: |
      curl -sS https://<link to decorator.js> -o dr-decorator.js
      echo "---------------------------------------------------------"
      node dr-decorator.js

- task: CmdLine@2
  continueOnError: true
  displayName: 'Clean Up DR Resources (DevOps Task)'
  inputs:
    script: |
      rm dr-decorator.js

- task: Kubernetes@1
  displayName: 'Kubernetes DR Logout (DevOps Task)'
  inputs:
    connectionType: 'Kubernetes Service Connection'
    kubernetesServiceEndpoint: 'DR-AKS'
    command: 'logout'

In the above YAML file example, we first log into the DR region AKS with the “Kubernetes@1” task (it will be used to deploy the application in the DR AKS in the second task). Then, in the second task, we download a custom script file “dr-decorator.js”.

As we’ll be configuring the decorator to run in all pipelines in the organization, this JS file runs custom checks to check if the DR process is required to run in the current pipeline or release.

Script Functions

The script functions on the following tasks:

• Get current pipeline details such as the YAML file referenced by the pipeline.
• Read and parse the YAML file to JSON to get all the necessary data like chart path and values file path to install the application in the DR region.
• Checks if the DR Process is required in the current pipeline or not by running the following custom checks:
  
  – isHelmDeployPipeline(): Checks if the pipeline includes a Helm deploy task or not.
  – isProdPipeline(): Checks if the pipeline is of the Production Environment.
  – isKindDeployment(): Check to ignore CronJobs to be deployed in the DR Region.

If all the mentioned checks return true, then the script runs the “helm install” command with the current pipeline’s chart path and values file to duplicate the prod environment. The YAML then removes the downloaded script file and logs out from the DR AKS.

Building and Publishing the Extension

To build the extension, we’ll need the “tfx” CLI installed on our local machine. Run the following command to build and publish the extension:

tfx extension publish --manifest-globs vss-extension.json --rev-version --share-with <Your organisation name> --token <PAT token with permission to publish extensions in VSS Marketplace>

A Personal Access Token (PAT) can be generated from Azure DevOps. The above command will keep the extension private and share it with your organization only.

You can checkout the script and other files in the below github:

GitHub - SidharathBansal/AKS-DR-Decorator

Installing the Extension in Azure DevOps

Once the extension is deployed, you need to go to Azure DevOps and install the extension globally into the organization by going into Organization Settings -> Extensions -> Explore Marketplace -> search for your extension and install it.

Sample Directory Structure

Here is an example of the directory structure for the project:

├── AKS-DR-Decorator
│   ├── decorator
│   │   └── AKS-DR-Decorator.yml
│   ├── images
│   │   └── decorator-logo.png
│   └── vss-extension.json

Sample Deployment Pipeline

Below is an example of a sample deployment pipeline setup:

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: HelmInstaller@0
  inputs:
    helmVersionToInstall: 'latest'

- task: HelmDeploy@0
  inputs:
    connectionType: 'Azure Resource Manager'
    azureSubscription: '<Your-Azure-Subscription>'
    azureResourceGroup: '<Your-Resource-Group>'
    kubernetesCluster: '<Your-Kubernetes-Cluster>'
    namespace: 'default'
    command: 'upgrade'
    chartType: 'FilePath'
    chartPath: '<Your-Chart-Path>'
    releaseName: '<Your-Release-Name>'
    overrideValues: '<Your-Override-Values>'

Pipeline Tasks After Decorator Injection

After injecting the decorator, the pipeline tasks are augmented to include the DR deployment steps:

Conclusion

By leveraging Azure Pipeline Decorator, we have significantly streamlined the DR deployment process for AKS at Bajaj FinServ Health. This innovative solution allows us to manage DR deployments centrally, ensuring consistency and scalability across hundreds of microservices. As a result, we have reduced the complexity and overhead associated with managing individual pipelines, enabling our teams to focus more on developing and improving our services.

Implementing Azure Pipeline Decorator for AKS DR is a testament to the power of automation and the flexibility of Azure DevOps. This approach can be a game-changer for any organization facing similar challenges in managing large-scale microservice environments.

For more information on authoring Azure pipeline decorators, refer to the official Microsoft documentation.

References

• Azure Pipelines Tasks GitHub Repository
• Microsoft Docs: Add a Pipeline Decorator

If you enjoyed this blog, I invite you to follow me and connect with me on LinkedIn.