Use Key Vault Secrets in any Azure Data Factory connection property

There is an undocumented way to reference Azure Key Vault secrets even when the UI doesn’t support it.

In a recent project we created a number of Azure Data Factory (ADF) pipelines which we needed to push through Dev / Test / Prod environments in a typical devops flow. Whilst this is pretty well documented, in practice there were a couple of things we lacked.

For instance, when connecting to Azure Databricks the UI wizard allows you to specify a key vault secret instead of hard-coding the PAT token you need to authenticate. This is great as it means in each environment (dev/test/prod) we have a key vault with the same secret name which stores the appropriate PAT token:

Specifying a key vault secret for the PAT token

However, we also wanted to dynamically populate the instance pool id from key vault too — but no luck in the UI:

Cannot use key vault to specify this (or many other) properties

Fortunately, using the code view it’s possible to specify a key vault secret which the UI doesn’t allow. To do this, open the code view of your connection and use the following snippet:

To Note:

• Making this change will disable the default UI wizard view, and will give you a code view instead.
• You can still use the wizard to test the connection:

Testing connection for updated json

That’s it! If you get a chance to try it out, or use this technique on other connection properties let me know :)