New CI/CD flow in Azure Synapse Analytics using Azure DevOps 10 step guide

With the latest release notes, the Automated deployment from any branch feature is now available.

New CI/CD flow with Automated deployment from any branch

The changes to the New CI/CD Flow are similar to Data Factory’s Continuous deployment improvements.

Current CI/CD flow

Below is the current Synapse CI/CD flow created to resemble the ms docs image.

1. Each user makes changes in their private branches.
2. Push to master/main isn’t allowed. Users must create a pull request to make changes.
3. Users must load the Synapse StudioUI and select Publish to deploy changes to Synapse Workspace and generate the ARM templates in the publish branch.
4. The DevOps Release pipeline is configured to create a new release and deploy the Synapse artifact template each time a new change is pushed to the publish branch.

New CI/CD flow

1. Each user makes changes in their private branches.
2. Push to master/main isn’t allowed. Users must create a pull request to make changes.
3. The Azure DevOps pipeline build is triggered every time a new commit is made to master/main. It validates the resources and generates a Synapse artifacts template as a build artifact if validation succeeds.
4. The DevOps Release pipeline is configured to create a new release and deploy the Synapse artifacts each time a new build is available.

Let’s get started

To experience this new flow, we created a multi-stage pipeline.
The multi-stage pipeline implements the following flows.(Deployment to Dev and Prod is omitted this time for simplicity.)
In addition, the multistage pipeline includes a practical implementation that temporarily changes the IP Whitelists in the Synapse Workspace during the CD stage.

1. Import Azure DevOps Repos from https://github.com/ryoma-nagata/SynapseCICD-template-workspace.git
   Import a Git repo into your project — Azure Repos | Microsoft Docs
2. Configure git setting.This step is explained in the following blog.
   CI CD in Azure Synapse Analytics Part 1 — Microsoft Tech Community
   Synapse dev git setting is below.

3. Publish template file and download TemplateParametersForWorkspace.json.

4. Upload TemplateParametersForWorkspace.json to main as TemplateParametersForWorkspace_stg.json.

Note: The link service connection string will be empty and must be set like nyc_tlc_green_sasUri in image. However, Default link service does not need to be set.

5. Create Azure DevOps Service Connection.
Service connections in Azure Pipelines — Azure Pipelines | Microsoft Docs

6. Add DevOps Service principal to Synapse Adminstrators for each Dev/Stg workspaces.
Quickstart: Get started add an Administrator — Azure Synapse Analytics | Microsoft Docs

7. Create DevOps envirionment STG.
This environment is used by DevOpsPipeline as the target environment for deployment tasks. Approval gates can be set as needed.

8. Create pipeline from “azure-pipelines-ci-cd-synapse-artifacts.yml .”

9. Edit Variables.
- ARTIFACTS_FOLDER: Synapse artifacts root folder when configure git integration.
- AZURE_RESOUCE_MANAGER_CONNECTION_NAME : Azure DevOps Service Connection
- TARGET_WORKSPACE_NAME_STG: Synapse STG workspace name.
- TARGET_RESOURCE_GROUP_NAME_STG: Resource group name that contain Synapse STG workspace.
For example, the following

10. Save and run .

Synapse CI/CD is now configured!
Then, Let’s Create feature branch and Pull request .

Create branch

Add new sql script from Synapse Studio Knowledge Center.

Commit and Create Pull request.

Merge PR.

The CI/CD pipeline is started.

After completion, your changes have been deployed in the STG environment!