Azure Data Factory build and deploy with new CI/CD flow using GitHub Actions

Azure Data Factory (ADF) now supports an ADFUtilities NPM package, @microsoft/azure-data-factory-utilities, to allow for build activities to run independently of the service. This post will show you how to setup a sample end to end workflow using Github actions with the new ci/cd flow.

A previous blog post in Technet, Azure Data Factory CI/CD with GitHub Actions, outlined how to setup GitHub Actions for ADF using the manual publish flow. The initial setup steps and requirements are the same up through to “Configure the GitHub Secrets”.

The updated flow:

Automated publishing for continuous integration and delivery — Azure Data Factory | Microsoft Learn

Updated guidance on creating the workflow to build and deploy the ARM template:

In addition to the ADFUtilities NPM package, there are marketplace actions published by Azure that make it even easier to implement the build and deploy job steps into your workflow. For validate and generate ARM templates we can use data-factory-export and for release there is data-factory-deploy.

Putting it all together, we can implement this workflow as follows:

1. Follow steps from Azure Data Factory CI/CD with GitHub Actions through to “Configure the GitHub Secrets”.
2. Add an additional secret that is the name of the Azure Data Factory deployment target.
3. Add a .yaml workflow definition file in the .github/workflows directory of your collaboration branch. It can have any name; in my repo I just named it after the branch:

Repo with GitHub workflow directory

Sample .yaml:

name: Deploy ARM template to Azure Data Factory

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Validate ARM template
        uses: Azure/data-factory-validate-action@v1.1.5
     

      - name: Export Azure Data Factory
        id: export
        uses: Azure/data-factory-export-action@v1.2.0
     
      
      - name: Publish ARM template
        uses: actions/upload-artifact@v2
        with:
          name: adf-armtemplate
          path: ${{ steps.export.outputs.arm-template-directory }}
          if-no-files-found: error
  
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Login via Az module
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true 
     
      - name: Download ARM template
        uses: actions/download-artifact@v3
        with:
          name: adf-armtemplate

      - name: Deploy resources
        uses: Azure/data-factory-deploy-action@v1.2.0
        with:
          resourceGroupName: ${{ secrets.AZURE_RG }} 
          dataFactoryName: ${{ secrets.AZURE_DATA_FACTORY }} 

          

For an explanation of the .yaml file, I phoned a friend:

ChatGPT code explanation curtesy of chat.openai.com

5. Add a branch protection rule to require PRs against the collaboration branch (at minimum):

6. Disable Publish (from ADF Studio):

Git configuration with Publish (from ADF Studio) Disabled

Now you have a baseline workflow for implementing the new ADF ci/cd flow using GitHub Actions. Instead of manually publishing updates to the ADF instance we initiate the build and deploy when committing the PR.

Initiating a PR from ADF Studio

Releases to higher environments can be managed by cloning the workflow deploy (and build) steps and replacing triggers and secret references as needed. The use of marketplace actions allows us the implement best practices for ADF deployment with a minimum amount of code.