Connect to the native Salesforce Data Cloud API through Named Credentials using a custom Auth Provider

Exploring a different approach to connect to (external) Data Cloud instances directly from Salesforce Flow or Apex.

14 min readSep 29, 2023

There are multiple ways to connect to Data Cloud directly from Salesforce. This allows you to build custom Data Cloud powered Flows or LWCs. The most common ways are using the Data Cloud Connect REST API and the Query API for (limited) queries; both part of the standard Salesforce REST API.

The CDP Classes in the Apex ConnectAPI Namespace allow you to connect to Data Cloud using Apex methods without the need for any API calls or connections. This will only work on the Home Org.

The Data Cloud Connect REST API is the best implementation option for most multi org implementations: you can simply connect to other Salesforce orgs declaratively and use named credentials to save yourself some customization overhead and follow best practices.
For the Home Org (An instance where Data Cloud is installed on the Same org as you're connecting to it) you can use the Apex Classes.

The Data Cloud Ingestion API and Data Graph API are not available through the Data Cloud Connect REST API. As far as I know all other functionality is available through the Connect API. So these are the two main use cases that require us the connect this way. See the below decision tree for clarity.

This article will explain in detail how to install and configure the Data Cloud Custom Auth Provider in combination with External and Named Credentials.

TL;DR The Github repository can be found here: https://github.com/jfwberg/lightweight-data-cloud-auth-provider

Important

Security is no easy subject: Before implementing this (or any) solution, always validate what you’re doing with a certified security expert and your certified implementation partner.

The process

To get access to the native Data Cloud API we first need to get an access token to for the Salesforce API using a JWT Exchange token flow. We then exchange the Salesforce access token for a Data Cloud access token.

Data Cloud returns an access token that is only valid with the Data Cloud API and and we use that as the OAuth Bearer Token for any subsequent calls to Data Cloud API endpoints using the Authorization Header.

Salesforce Data Cloud Native API Token Exchange Process

Why a custom authentication provider?

The process is not complicated; we can achieve the Salesforce to Salesforce connections already OOTB with Named Credentials, but we cannot do the token exchange with the Data Cloud API OOTB so we have to built something custom that can combine all steps and act as a single authentication flow. A custom authentication provider is the correct way to do this, so it can be used with Named Credentials.

The great advantage of using named and external credentials is that Salesforce takes care of the token process: Retrieving a new token, storing tokens securely and automatically fetching a new token if a token has expired.
Refresh tokens are not supported in this flow; when a token is expired it will fetch a new token using the full flow. Having Salesforce take care of all the heavy lifting saves you building something yourself resulting in a more secure and robust process.

Developer Note: We cannot use a named credential for the first step because it’s not possible to get the token value except at run time using a merge variable. The Salesforce Token needs to be URL encoded when requesting the Data Cloud Token and this cannot be achieved through code. It’s a little annoying, but I am glad to see tokens are handled securely.

Example — Query

In this example we call a Named Credential Called “HomeOrgDataCloud” from Apex and execute the most simple SQL query I could think off “SELECT 1”.

I can now simply run an Apex snippet to execute a query on Data Cloud without a single worry about the token process.
Note that the content-type is always required or you will get an error response.

 HttpRequest request = new HttpRequest();
 request.setEndPoint('callout:HomeOrgDataCloud/api/v2/query');
 request.setHeader('Content-Type','application/json');
 request.setMethod('POST');
 request.setBody('{"sql" : "SELECT 1"}');
 HttpResponse res = new HTTP().send(request);
 System.debug(res.getStatusCode());
 System.debug(res.getBody());

This results in the following response:

{
    "data": [
        [
            1
        ]
    ],
    "startTime": "2023-09-29T10:47:29.249158Z",
    "endTime": "2023-09-29T10:47:29.793916Z",
    "rowCount": 1,
    "queryId": "20230929_104729_07329_htgtn",
    "done": true,
    "metadata": {
        "_col0": {
            "type": "INTEGER",
            "placeInOrder": 0,
            "typeCode": 4
        }
    }
}

Example — Metadata

In this example we call the metadata endpoint

HttpRequest request = new HttpRequest();
request.setEndPoint('callout:HomeOrgDataCloud/api/v1/profile/metadata');
request.setMethod('GET');
request.setHeader('Content-Type','application/json');
HttpResponse res = new HTTP().send(request);
System.debug(res.getStatusCode());
System.debug(res.getBody());

This results in the following response:

{
    "metadata": [
        {
            "fields": [
                {
                    "name": "segment_1sg8d000000sZsvBAE__c",
                    "displayName": "Segment 1sg8d000000sZsvBAE",
                    "type": "NUMBER"
                }
            ],
            "indexes": [],
            "category": "Profile",
            "name": "Individual__dlm",
            "Etc.." : "Etc.."
        }
    ]
}

As you can see, it’s now really easy to call our Data Cloud APIs in a secure and easy manner using a named credential.

Step by Step setup

To make clear the difference between the Orgs that we are going to connect using I’ve outlined the difference:

Salesforce Org

This Org is doing the callout to the external Data Cloud Org(s).
This Org contains the custom Auth Provider Package
This Org is where we will set up the Named/External Credentials
This Org contains the Certificate that is used for signing the JWT when connecting to the Data Cloud Org(s). that is stored in the Salesforce certificate store.
In my example I am going to call this "Hub Org"

Data Cloud Org(s)

This Org has Data Cloud provisioned on it
This Org is where we will set up a Connected App
The Data Cloud Org can be the same org as the Salesforce Org. The steps will and the My Domain URL will the same.

Pre-requisites

The Salesforce Org My Domain URL (https://hub_org.my.salesforce.com)
The Data Cloud Org My Domain URL (https://dc_org_01.my.salesforce.com)
The Data Cloud Instance URL(https://random-id.c360a.salesforce.com)
The Data Cloud Org needs a Certificate with public and private key that is used for signing the JWT. This should be a CA signed certificate but you can use a self signed certificate for testing purposes. This should be saved in the Salesforce certificate store.
Have the public certificate file saved on your machine, as we need this later.
The Data Cloud Org needs to have an Integration User with Data Cloud Access Setup.
Optionally: Install the Auth Provider Util on the Salesforce Org, for easy error reporting and "per user" management.

01 :: Salesforce Org :: Preparation Steps

Import a Signing Certificate into Salesforce or create a self signed one for testing purposes.
Note down the Certificate API Name and download the certificate file to your machine. For certificate import troubleshooting see the paragraph at the end of this document.
The (dependent) packages can be installed using the information found in the GitHub repositories or blog posts:
- Lightweight — Apex Test Util v2 (v2.4) (Install) (Blog 01 / Blog 02)
- Lightweight — REST Util (v0.11)(Install)
- Lightweight — Data Cloud Auth Provider (v0.5)(Install) (Blog)
The Auth Provider Util is optional but recommended for easy debugging.
- Lightweight — Auth Provider Util (v0.12)(Install)(Blog)
If you have not installed the packages but deployed all repositories manually. You'll have to remove the "utl" and "lwt" namespace references from the code.
Now you have your certificate set up and all required packages installed, we are going to setup the Data Cloud Org.

02 :: Data Cloud Org :: Create Connected App Permission Set

To allow our integration user to connect through a connected app to the Data Cloud API, we need to setup a permission set.

In Setup go to Users >> Permission Sets and Click the “New” button
In the “Label” put “Hub Org Connected App Access”
The “API Name” should automatically populate
The “License Type” dropdown can be kept to “None”
Press “Save”
Assign this permission set to the Integration User now; forgetting this might cause head scratching later.
This is all for now, there is no need to assign any permissions

03 :: Data Cloud Org :: Setup the Connected App

The Salesforce Org called Hub org will connect to this Data Cloud Org. The first step is setting up the “App” that “connects” to this org using a Connected App.

In your Data Cloud Org go to Setup >> Apps >> App Manager
Click New “Connected App”
For “Name” put “Hub Org”, “Api Name” “Hub_Org” and the admin’s email address in “Contact Email”
Check “Enable OAuth Settings”, additional settings will appear
In the “Callback URL” put your Salesforce Org’s My Domain Url and the Salesforce default OAuth 2.0 callback endpoint
i.e. “https://hub_org.my.salesforce.com/services/oauth2/callback”
Check “Use digital signatures” and use the “Choose File” button to upload the Certificate you downloaded from the Salesforce Org in step 1.1
In “Selected Scopes” select:
Perform ANSI SQL queries on Data Cloud data (cdp_query_api)
Manage Data Cloud profile data (cdp_profile_api)
Perform requests at any time (refresh_token)
Manage user data via APIs (api)
You can select additional Data Cloud scopes if you require additional APIs. The Data Cloud API return a scope error if a scope is missing for the API you’re calling.
Press “Save”. If you’ll get a warning message press “Continue”. It might take 10 minutes for your app to properly work. Usually it’s quicker, but keep this in mind when testing: Take ample time before connecting.
You will be redirected to your Connected App.

04 :: Data Cloud Org :: Setup the Connected App Policies

In your newly created Connected App click the “Manage” button
In the policy screen click the “Edit Policy” button
Under the section “OAuth Policies”, in the “Permitted Users” Dropdown select “Admin approved users are pre-authorised”
Press “Save”
In the policy screen, scroll down to the “Permission Sets” section. Click “Manage Permission Sets”. Select the “Hub Org Connected App Access” permission set you created in Step 2
Go back to your Connected App, this has to go through Setup >> Apps >> App Manager. In the list of apps select “View” from the options arrow on the right side next the “Hub Org” App.
Click the button “Manage Consumer Details”, you might need to go through Multi-Factor Authentication to see these details, this is expected.
Copy the “Consumer Key”, you will NOT need the “Consumer Secret”. Store the consumer somewhere in a temporary note, we’ll need this in a later step.
Close the window. Always keep consumer data safe!
Always discuss the security settings with your company’s security team and follow the principle of least access.

05 :: Salesforce Org :: Setup the Remote Site Setting(s)

In order to make API call-outs to the Data Cloud Org’s token endpoint, we must set up a remote site setting for the token endpoint. The endpoint is called from the Auth Provider Apex class, so white listing is required.

1. Go to Setup > Security > Remote Site Settings, Click New

2. Populate the Remote Site URL field with the Data Cloud Org Base URL and set a Name and Description

3. Press Save

05 :: Salesforce Org :: Setup the Auth. Provider

In my example I am going to connect to an Org called “DC_ORG_01”; this my test data Cloud Org that I will use throughout this example. Replace this with your own Org Name. This is purely an illustrative naming convention.

In Setup > Auth. Providers, Create a new Auth. Provider Using the DataCloudAuthProvider class as the provider type.
Populate the “Execute Registration As” field first, this is a mandatory field that is not marked as mandatory and will reset the entire form if you forget it. The registration is not used.
Make sure that Name, URL Suffix and Auth Provider Name fields all have the same value.
Populate the fields in the the Auth. Provider.
- The Connected App Id with the Id you noted down in Step 4.8
- The integration user name should be the user you’ve assigned the permission set to in Step 2.6
- The certificate name is the API name from the certificate in Step 1.1
- The my domain URL is the Data Cloud Org My Domain URL
- Optionally fill in the Data Space field with the desired data space API name. If you only have the default data space, keep this field blank.
Triple check you put in all mandatory fields: If you have forgotten one, you’ll have to re-do them all.
If you have the optional Auth Provider Util package installed on the Salesforce Org you can check Enable Error Logging and Per User Mode. Per User Mode is out of scope, but is done as explained in the Util. For error logging this is a very useful tool though.
!! If not installed leave all checkboxes blank !!

06 :: Salesforce Org :: Setup an External Credential

Your Auth Provider is now ready for testing. The next step is to create an External Credential that authenticates to the token endpoint(s) using the Auth Provider.

Go to setup > Security > Named Credentials and click the External Credentials tab.
Click New. Set A Label and a Name, in my Case “DC_ORG_01".
Set Authentication Protocol to OAuth 2.0.
Set Authentication Flow Type to Browser Flow. The Scope field can be left blank. This is overwritten by our Auth. Provider Settings.
Select your created Auth. Provider from the Auth Provider Picklist. In my case “DC_ORG_01”.
Press “Save”.

Once saved, scroll down to the “Principals” section
Click “New”, for a name give it something like “DC_ORG_01_EXT_CRED_ACCESS”
Scope can be left blank and the sequence can be kept default
Press “Save”
Leave the external credential open, we have to make a quick side step

07 :: Salesforce Org :: Create a Permission Set for the External Credential

External Credentials require a Permission Set to provide access to the external credential. It’s best practice to create a separate Permission Set s for each Extern Credential to keep a strict separation and stick to the least access principle.

In Setup > Users > Permission Sets, Click New
Set a Label and an API Name, I am going for “DC ORG 01 External Credential Access”. And press “Save”.
Scroll down to the section called “External Credential Principal Access”
Assign the “DC_ORG_01_EXT_CRED_ACCESS” principal from “Step 6.2” to the permission set and press “Save”.
Assign the Permission Set to the testing user, most likely the user you’re logged in with.

08 :: Salesforce Org :: Connect your External Credential

You have now finished setting up the External Credential and the Auth Provider. The moment of truth…

Go back to your External Credential.
Click on the arrow button next to the “DC_ORG_01_EXT_CRED_ACCESS” principal you created in Step 2.6
Click Authenticate, Salesforce now call the token endpoint and execute the logic from the Apex class to get a token. It will redirect you to the same page (This is the redirect URL that is auto generated in the class.)
If all goes well you get a green toast method saying you connected successfully. Congratulations! You’re now successfully authenticated.

09 :: Salesforce Org :: Create a Named Credential

Once you have successfully authenticated your External Credential, there is one more step required to use it from Apex or Flow to call an API: We need a Named Credential that uses the External Credential.

1. Go to setup > Security > Named Credentials and click the Named Credentials tab

2. Click New (Note we are NOT creating a legacy named credential, we’re creating a modern one)

3. Give your credential a Label and a Name, I am going to be original and call it “DC_ORG_01”.

4. Populate the URL field with the base URL of the Salesforce Data Cloud API (Not the token endpoint). It will look like something like:
“https://g-yt1yjzmfrti323bdgdscgy.c360a.salesforce.com”

5. Select your “External Credential” you created in Step 6, i.e. DC_ORG_01

6. Make sure “Generate Authorization Header” is selected, this is by default

7. Optionally you can select a namespace and a network if you use private connect. In my example I allow my “utl” and “lwt” namespaces, but be aware to what managed packages you give access.

8. Press “Save”

10 :: Salesforce Org :: Testing

That is it… We’re all done and set up now. So we can test. The code has already been shown at the top but I’ll reapeat it here.

Open the “Developer Console” or any IDE of choice that can execute Apex Code Anonymously.
Execute the below code block that uses the Named Credential syntax with our Named Credential’s name “callout:DC_ORG_01/api/v2/query”
In the test I have added the query endpoint. Please note that the named credential name is case sensitive an there should be no whitespace around it.

HttpRequest request = new HttpRequest();
 request.setEndPoint('callout:DC_ORG_01/api/v2/query');
 request.setHeader('Content-Type','application/json');
 request.setMethod('POST');
 request.setBody('{"sql" : "SELECT 1"}');
 HttpResponse res = new HTTP().send(request);
 System.debug(res.getStatusCode());
 System.debug(res.getBody());

If there are no errors and the code executes successfully, you are good to go.

Although it uses apex in the Auth Provider, you’ve created the actual integration with clicks, not code. Named Credentials can also be used with Flows and External Services.

Common Configuration Errors

A mistake is easy to make so here are the most common ones:

Not created a Remote Site Setting to the Data Cloud Org on the Salesforce Org
Not assigning the Connected App Permission Set on the Data Cloud Org to the Integration User
Not assigning the Permission Set on the Salesforce Org
Wrong certificate name (Note this is Case Sensitive, you get a “System.NoDataFoundException: Data Not Available” exception if the Certificate API Name is wrong)
The Integration User does not have access to Data Cloud. Make sure that the Integration User you have selected has access to Data Cloud. Commonly you might see something like: {“error”:”access_denied”,”error_description”:”end-user denied authorization”}
Make sure you have the Data Cloud permissions sets assigned and if you want to be sure, create an initial test with an Administrator profile before switching to a Minimum Access Profile.

I don’t want to go to deep into debugging but a few pointers:

Set the trace flag on the executing user
Clean your debug logs in setup, execute the code and validate the logs
Alternatively, open your developer console to stream the logs
Alternatively, open your developer console to stream the logs

Steps to import a certificate store (JKS) in a Scratch Org when getting “Data Not Available” error message on import

In some cases there is a bug in the certificate import that gives an error if you try to import a JKS It says “Data Not Available”.

I have this issue in all my scratch orgs. There is a simple way to resolve this.

Go to setup > certificate and key management
Create a self signed certificate
Go to Setup >> Identity provider
Click enable Identity Provider and select the self signed certificate you just created and press save
Press the disable button, as we don’t really need it
Go back to Setup > Certificate and Key Management and try to “import from keystore” again, it should work now.

Final note

At the time of writing I work for Salesforce. The views / solutions presented here are strictly MY OWN and NOT per definition the views or solutions Salesforce as a company would recommend. Again; always consult with your certified implementation partner before implementing anything you’ve read on the internet.