Azure Databricks Unity Catalog — Part 3: Automate Unity Catalog set up using Terraform

Databricks & Terraform

This is Part 3 of series — Azure Databricks Unity Catalog — up and running; we walkthrough how to automate the provisioning of Azure Databricks Unity Catalog Metastore, and also show examples documented here in action, please refer to the original guide for step-by-step explanation:

Terraform Registry

We have simply put the code blocks of documented guide together into an example deployment, you can find the example here:

terraform-databricks-examples/examples/adb-uc/stage_3_spn_deploys_uc at main ·…

For each UC metastore, we recommend using dedicated storage account. Note that in storage.tf we have configured the networking restrictions to this storage account, similar to part 2 tutorials but this time it’s all through Terraform.

resource "azurerm_storage_account" "unity_catalog" {
  name                     = "${local.prefix}ucmetastore"
  resource_group_name      = data.azurerm_resource_group.this.name
  location                 = data.azurerm_resource_group.this.location
  tags                     = data.azurerm_resource_group.this.tags
  account_tier             = "Standard"
  account_replication_type = "ZRS"
  is_hns_enabled           = true

  network_rules {
    default_action = "Deny"
    bypass         = ["None"]
    private_link_access {
      endpoint_resource_id = azurerm_databricks_access_connector.unity.id
    }
  }
}

Since we’ve blocked all public Internet access to this storage account, our machine running Terraform for deployment will not be able to modify this storage account unless we add our Terraform machine’s IP into whitelist.

After running terraform init and terraform apply we can see the deployed resources (red box):

UC related resources

Now we’ve completed the 3 articles in this mini series and if you’ve followed along, now you are ready to populate your UC metastore and start access management on securable (catalogs/schemas/tables). We’ll add more tutorials to this series in the future on UC administration and operations.