Using Azure Databricks Credential Pass-through to Access Data Stored in Azure Data Lake

Databricks Credential Pass-through allows you to access data stored in Azure Data Lake without passing any credentials explicitly. It uses the identity that you used to login to Azure Databricks.

Credential Pass-through works when following conditions are met.

1. Should have the following spark configuration set in your notebook

#Config for Gen2 Lake
spark.conf.set("fs.azure.account.auth.type", "CustomAccessToken")
spark.conf.set("fs.azure.account.custom.token.provider.class", spark.conf.get("spark.databricks.passthrough.adls.gen2.tokenProviderClassName"))

2. Should have the following setting ON on Databricks Cluster

Credential Passthrough setting on Databricks Cluster

3. User who runs the notebook should have Read / Execute permissions on the entire path in Data Lake Gen2 including filesystem. For example, if user wants to read a file under path /Oregon/Portland/Data.txt the following permissions are needed

Screenshot from MS Documentation on ACL permissions

For more information on setting up ACL permissions refer to Microsoft Documentation