Azure Data Factory Build and Release Pipeline (ADF CI/CD) through Azure DevOps

Pre-requisite:

Access to Azure portal data factory

AzureDevOps with code, build and release pipeline

Data factory code should be available in the Azure DevOps repository

Get Started

Plan: After creating dev_adf in the Azure portal same changes should deployed to the test environment through the release pipeline

Login to Azure Portal

create dev_adf in the dev resource group

Go to the marketplace and search data factory

Click on Create

Goto Resource created adf add some pipeline and publish

Now time to go to Azure DevOps to see and create a build pipeline Before that we have to add power shell scripts to stop and start and delete unused resources in a release pipeline

To do that go to Azure DevOps of ADF repository with a branch called adf_publish

Create one Scripts folder and upload RemoveOrphanedADFResources.ps1 and SetADFTriggersState.ps1

How to get Scripts:
Fortunately for us, Jeff Moss has a public GitHub Repo with a bunch of great PowerShell scripts (among other things, such as IaC, Databricks, SQL, Bicep, etc. — I highly recommend to check it out!)

Click on the new pipeline

click on Use the classic editor

Select Team Project, Repository and branch will be adf_publish

Click on Continue

Select Empty Job

Click on + Symbol in Agent job 1

Search Publish build artifacts

Click on Add

Save

Click on the Triggers Tab

Checked on Enable continuous integration

Select the branch as adf_publish

Go to the Options tab and add the Build number format

Goto Variables Tab add Majo and Minor Variable used for Build number format

After saving it click on the Run pipeline to create the first build

Click on Run

Now Artifact is available for the Release Pipeline

Let Go for Release Pipeline

Goto Release and add new Release Pipeline

Click on Empty job

Will follow 4 Steps to complete the release of ADF Release to test the environment

1. Stops Triggers before Deploy ARM template for ADF
2. Deploy the latest ARM Template with incremental Deployment mode
3. Delete the Orphaned Resouce
4. Restart Triggers (which will only restart previously started Triggers if any of them are disabled before deployment, it won’t start)

Step 1: Add Azure PowerShell

-DataFactoryName “$(DataFactory)” -DataFactoryResourceGroupName “$(ResourceGroup)” -State “Stop” -ReleaseIdentifier $(Build.BuildNumber)

Scripts Arguments can be set in the variable for different environment

Click on Scripts Argument to add all required variable

Goto variable sections and add variable values

Step 2: Add ARM Template

Deployment mode should Incremental

Override template parameters: Override required parameter

Step 3: Delete the Orphaned Resouce

Scripts Arguments

-DataFactoryName “$(DataFactory)” -DataFactoryResourceGroupName “$(ResourceGroup)” -armTemplate $(System.DefaultWorkingDirectory)/_ADF-Ci/drop/ARMTemplateForFactory.json

Step 4: Restart Triggers

Script Arguments

-DataFactoryName “$(DataFactory)” -DataFactoryResourceGroupName “$(ResourceGroup)” -State “StartPriorEnabled” -ReleaseIdentifier $(Build.BuildNumber)

The final Release Pipeline Looks like