ADF PARAMETERS DEPLOYMENT FOR MULTI-ENVIRONMENT — AZURE DEVOPS

Published in

Globant

4 min readJan 18, 2021

INTRODUCTION

Azure Data Factory is the platform that solves such data scenarios. It is the cloud-based ETL and data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale. Simply said, it is a service that enables you to move data from one service to another in a split-second.

Azure Data Factory = Data Migration + Analytics

As the world is moving into the cloud and big data; data integration and migration will remain essential elements for organizations across industries. ADF can help you address these two concerns efficiently by enabling you to focus on your data and allow you to schedule, monitor and manage your ETL/ELT pipelines with a single view.

Here are some reasons why the adoption of Azure Data Factory is on the rise: Drive more value, Improve business process outcomes, Reduce overhead expenses, Better decision-making, Increase business process agility.

In this blog, we are going to discuss the challenge around handling ADF parameters while deploying through the Azure DevOps CI/CD pipeline and a solution to address the challenge. The challenge is that you can’t see the ADF Data Set and Pipeline parameters in the Azure DevOps CI/CD pipeline so you can’t change the parameter value at run time.

PRE-REQUISITES

Basic knowledge of ADF and Azure DevOps CI/CD pipeline (Ref. https://docs.microsoft.com/en-us/azure/data-factory/introduction)
Azure DevOps Project
Connect source control in the Azure data factory (Ref. https://docs.microsoft.com/en-us/azure/data-factory/source-control)

ADF PARAMETER CHALLENGE

A. For demonstration purposes, I have created one Blob-linked service connection with the ADF instance.

B. Created one data set.

As you can see in the File path section @dataset().SourceFilePath. This path is dynamically added and it has come from the Parameters tab.

C. Create one ADF pipeline. It’s a copy data task.

D. Once published ADF pipeline, It will automatically create one adf_publish branch with the ARM templates.

The ARM template parameter file will contain only the linked service parameters, not Dataset and pipeline parameters.

E. Now create a Release pipeline for the different environments. (eg: QA). But in the release pipeline parameter section, you can only see the linked service parameter.

# If we need to change the parameters of the Dataset and ADF pipeline we can’t change them in Azure DevOps because by default it takes only the linked service parameters.

SOLUTION

In order to get the Dataset and ADF pipeline parameters value in the Azure DevOps release pipeline.

Create a custom JSON template in the root folder of the collaboration branch with the same name arm-template-parameters-definition.json as you can see in the image below the master is the collaboration branch and created a custom JSON template in the root path.

Note:- Once you have committed your custom JSON parameter template, then refresh the ADF pipeline (In the Azure portal) and publish the ADF pipeline.

#Now you can see all the Dataset and pipeline parameters in the JSON template.

In the below image you can see all the parameters in the Azure DevOps pipeline and you can change the parameters from one environment to another. Ex. Dev to QA.

CONCLUSION

Although Microsoft has suggested using an override template to achieve this, however, they have not provided the changes to be done in ADF. We have demonstrated the changes to be made in ADF through this article. We have achieved it by forming a custom JSON template and publishing the ADF pipeline.