A DevOps Approach to Marketing Cloud Deployments

Many enterprises have adopted Salesforce Marketing Cloud to improve their digital marketing automation and build customer relationships. For Marketing Cloud users, migrating data extensions across business units is a task that it too often completed manually. To help you avoid this inefficient approach, this post describes an alternative based on DevOps principles that you can use to standardize and automate the deployment of data extensions across business units in a repeatable fashion.

Setting up Marketing Cloud to work with source control

The approach covered in this post is for a single organization with separate business units that are used to isolate pre-production and production workloads.

Here are some prerequisites before getting started with the next steps.

• Set up a new package in Marketing Cloud. In Setup, click Apps and then Installed Packages. Click New and type a name and description. Under Components, click Add Component, turn on API Integration, and select Server to Server Integration as the Integration Type. This type of integration uses OAuth 2.0 authentication to obtain an access token. Enable read/write permissions in the Data Extensions section to use REST and SOAP APIs.
• Under the component details, locate the client ID and client secret, if applicable. Use these credentials with the Authentication Base URI to get an access token to authenticate your API requests.

You’ll be using Git with the Gitflow workflow to manage three business units: TEST, QA, and PROD. This workflow assigns specific responsibilities to named branches in the repository and defines how and when they should interact.

In Git, there is one branch for every business unit. The relations are:

• Branch Develop >> BU TEST
• Branch Release >> BU QA
• Branch Master >> BU PROD

Creating a data extension and committing it to source control

Marketing Cloud offers a SOAP API for working with data extensions. In this approach, you use this API instead of using the UI because it enables you to version the payloads of your API calls in your Git repository. To create, update, retrieve, or delete a data extension you can use this service:

https://[YOUR_SUBDOMAIN].soap.marketingcloudapis.com/Service.asmx

Suppose, for example, that you want create a new data extension in the business unit you are using for testing. You would use an XML payload like this:

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
   <s:Header>
      <a:Action s:mustUnderstand="1">Create</a:Action>
      <a:To s:mustUnderstand="1">https://{{et_subdomain}}.soap.marketingcloudapis.com/Service.asmx</a:To>
      <fueloauth xmlns="http://exacttarget.com">{{et_token}}</fueloauth>
   </s:Header>
   <s:Body xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <CreateRequest xmlns="http://exacttarget.com/wsdl/partnerAPI">
         <Objects xsi:type="DataExtension">
            <Client>
               <ID>{{et_mid}}</ID>
            </Client>
            <CustomerKey>Bounces_And_Helds_History_{{et_mid}}</CustomerKey>
            <Name>Bounces_And_Helds_History</Name>
            <CategoryID>{{et_categoryID}}</CategoryID>
            <!-- Create folder in the target BU and select and insert categoryID  -->
            <IsSendable>false</IsSendable>
            <Fields>
               <Field>
                  <CustomerKey>BHH_SubscriberKey_{{et_mid}}</CustomerKey>
                  <Name>SubscriberKey</Name>
                  <FieldType>Text</FieldType>
                  <MaxLength>256</MaxLength>
                  <IsRequired>true</IsRequired>
                  <IsPrimaryKey>true</IsPrimaryKey>
               </Field>
               <Field>
                  <CustomerKey>BHH_EmailAddress_{{et_mid}}</CustomerKey>
                  <Name>Email</Name>
                  <FieldType>EmailAddress</FieldType>
                  <MaxLength>254</MaxLength>
                  <IsRequired>false</IsRequired>
                  <IsPrimaryKey>false</IsPrimaryKey>
               </Field>
               <Field>
                  <CustomerKey>BHH_InsertDate_{{et_mid}}</CustomerKey>
                  <Name>InsertDate</Name>
                  <FieldType>Date</FieldType>
                  <DefaultValue>GETDATE()</DefaultValue>
                  <IsRequired>false</IsRequired>
                  <IsPrimaryKey>false</IsPrimaryKey>
               </Field>
            </Fields>
         </Objects>
      </CreateRequest>
   </s:Body>
</s:Envelope>

This payload includes the following variables:

• Subdomain
• Token
• Marketing Cloud Id (MID)
• CategoryID

In a manual run, the developer must update the variables with the values from their Marketing Cloud instance before executing the code above. When the call executes successfully, the new data extension is created in the business unit.

To version this payload in Git:

1. Store the payload in a new file (for example: NAME_OF_DATA_EXTENSION.txt) in a folder of the repository. For instance, in this scenario, you can store Bounces_And_Helds_History.txt into a path: MY_REPO_GIT/Marketing_Cloud/Data_Extensions/Bounces_And_Helds_History.txt
2. Execute these commands in git:

git checkout develop #switch to branch develop (branch of BU Test)
git add Bounces_And_Helds_History.txt #promote pending changes in the working directory
git commit -m "added new payload of Data Extension Bounces_And_Helds_History"
git push #push the commit in the global repo

Automating the deployment of your changes across business units

Although the example above made use of source control, it still required manually deployment of the data extension. It is possible, however, to avoid manual execution and directly make use of DevOps automation. In fact it is possible to build a pipeline that automates the versioning and the releases of your data extensions across different environments using Git, Jenkins and, of course, the Marketing Cloud APIs. This flow is represented here:

Let’s take a closer look at the workflow in detail for each step in the diagram above:

1. During the develop phase, a new data extension is created.
2. If needed, the data extension can be retrieved later via API.
3. After testing, the data extension can be pushed to the Develop branch.
4. A pull request can be initiated to push the data extension to the QA business unit.
5. The Release Management team can verify and approve the pull request.
6. With Jenkins, an automated workflow can calculate the delta between the current and previous commit and deploy the data extension to the QA business unit. If you’re not familiar with it, Jenkins is an open-source continuous integration/continuous delivery and deployment (CI/CD) automation DevOps tool written in the Java programming language. It is used to implement CI/CD workflows, called pipelines. The Jenkins pipeline is written using Groovy and Bash and its main purpose in this use case is to call a Java app written to encapsulate all the logic of the integration with Marketing Cloud.
7. When the code is ready for production, the Release Management team executes a merge in Master.
8. The same Jenkins pipeline is executed on the Master branch and the data extension is installed in the Production business unit.

Since the pipeline is fully automated, you can use the same pipeline by switching between pre-production and production business units.

Conclusion

This post covered one way to apply a source-driven DevOps approach to deploy data extensions across business units. This approach can help you automate release deployments in a repeatable and agile manner, minimize manual processes, reduce errors, and deliver business capabilities more quickly.