Trigger an ADF Pipeline (with parameters) using azure-sdk-for-go
Last few weeks I was busy working on a very interesting POC at work. This led me to delve deep into the robust capabilities of the azure-sdk-for-go package. As a developer navigating the Azure cloud ecosystem, I was intrigued by the potential of harnessing the capabilities of Go to interact seamlessly with various Azure services. The azure-sdk-for-go package emerged as a reliable companion, offering a plethora of tools and functionalities to streamline the integration process. In a series of blog post, I’ll share insights into my exploration of the azure-sdk-for-go, how to leverage this package to interact with various services.
There will be a series of blogs that will cover these specific tasks:
- Getting a fileshare used and left storage space.
- How to trigger a ADF pipeline using REST API
- How to get a Blob size
In this blog we will cover how to trigger a ADF pipeline using a REST API.
Before delving into the intricacies of triggering Azure Data Factory pipelines, it’s essential to grasp the foundational concepts of the Azure SDK for Go. This sophisticated package provides developers with a comprehensive set of libraries, allowing seamless interaction with various Azure services. Leveraging the power of Go, a language known for its simplicity and efficiency, the SDK simplifies complex tasks, including authentication, resource management, and, crucially for our purposes, the initiation of workflows within Azure Data Factory. Whether you’re a seasoned Go developer or new to the language, the Azure SDK for Go opens up a world of possibilities for orchestrating and automating your cloud workflows. In the subsequent sections, we’ll explore how to harness this toolkit to interact with the Azure Data Factory’s REST API, demonstrating the flexibility it brings to your data pipeline management.
package main
import (
"context"
"fmt"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/datafactory/armdatafactory"
)
func main() {
// Provide your Azure credentials
clientID := "Your_Client_ID"
clientSecret := "Your_Client_Secret"
tenantID := "Tenant_ID"
// Create a credential using ClientID, ClientSecret, and TenantID
cred, err := azidentity.NewClientSecretCredential(tenantID, clientID, clientSecret, nil)
if err != nil {
fmt.Printf("Failed to create credential: %v\n", err)
return
}
// Provide Azure Data Factory details
subscriptionID := "Your_Subscription_ID"
resourceGroup := "Your_rg"
dataFactory := "Factory_name"
pipelineName := "PipelineName"
parameters := map[string]interface{}{
"Parameter1 Key": "Parameter1_value",
}
// Create an ADF client
ctx := context.Background()
client, err := armdatafactory.NewPipelinesClient(subscriptionID, cred, nil)
res, err := client.CreateRun(ctx, resourceGroup, dataFactory, pipelineName, &armdatafactory.PipelinesClientCreateRunOptions{
ReferencePipelineRunID: nil,
IsRecovery: nil,
StartActivityName: nil,
StartFromFailure: nil,
Parameters: parameters,
})
if err != nil {
println("Err in triggering")
}
runId := res.RunID
fmt.Println("Pipeline run ID:", *runId)
fmt.Println("Pipeline triggered successfully!")
}
This Go code is an example of using the Azure SDK for Go to trigger an Azure Data Factory (ADF) pipeline run with some parameters. The Azure SDK for Go provides a set of packages and modules that allow you to interact with Azure services in Go applications.
Here’s a breakdown of the key components in the code:
Azure Credentials:
clientID: Azure Active Directory (AD) client ID.clientSecret: Azure AD client secret.tenantID: Azure AD tenant ID.
These credentials are used to create an Azure AD client secret credential, which is then used to authenticate the Go application with Azure.
Azure Data Factory Details:
subscriptionID: Azure subscription ID.resourceGroup: Azure resource group where the Data Factory is located.dataFactory: Azure Data Factory name.pipelineName: Name of the Azure Data Factory pipeline to be triggered.parameters: Parameters to be passed to the pipeline run. In this example, a map of parameter key-value pairs is provided.
Creating Azure SDK Client:
azidentity.NewClientSecretCredential: Creates a new Azure AD client secret credential.armdatafactory.NewPipelinesClient: Creates a new Azure Data Factory client for managing pipelines.
Triggering Pipeline Run:
client.CreateRun: Initiates the execution of the specified pipeline with the provided parameters.ctx: Background context.resourceGroup,dataFactory,pipelineName: Identifiers for the Data Factory and pipeline.&armdatafactory.PipelinesClientCreateRunOptions: Options for pipeline run creation, including parameters.
Handling the Response:
res, err := client.CreateRun(...): The method returns a response and an error.- If there’s an error during the pipeline run creation, it prints an error message.
- If successful, it extracts the run ID from the response and prints a success message along with the run ID.
Note — The Client Id and Client Secret is of the service principle that has appropriate access to trigger a ADF pipeline.
Hopefully, this blog gave you an understanding of how you can trigger an ADF pipeline using azure-sdk-for-go. If you have any questions or comments, please feel free to leave them below. Your feedback is always appreciated.
