Quickly Enable Unity on your Azure Workspace

Author: Jason Drew

Intro:

As you probably already know, Unity is the future of Databricks. It gives you a consistent governance method across all your assets in your Delta Lakehouse as well as enhanced search, lineage, auditing and sharing capabilities. Not only that, but many upcoming features such as Serverless Workflows will require Unity.

While not entirely true, many features will require Unity going forward.

This isn’t meant to be an all inclusive Production implementation guide, rather a quick start to show how to enable Unity in your workspace in the simplest way possible. Also, while the concepts maybe similar in other clouds, this is specific to Azure.

Personally I’m a very visual learner, so while these steps are in the documentation, I thought it would be useful to actually show each step visually as best I could. To help facilitate this, I spun up a fresh Azure Workspace from scratch so everything is a clean and easy to follow.

Prerequisites:

1. You must have Databricks Account Admin access. At first, only your AD Global Administrator role will have access your Databricks Account Admin interface, but they can grant other users the Account admin role.
2. You need an ADLS Gen 2 storage account with Hierarchical namespace Enabled within the same Region as your Workspace for your “Unity Metastore”. I put this in quotes because this isn’t actually where metadata is stored, rather just the default location for MANAGED tables. Also, while multiple Workspaces within an account can share the same Metastore, this is limited to Workspaces within the same Region.
3. You need an Access Connector for Azure Databricks that is assigned the Storage Blob Data Contributor role to a Scope that includes the Storage Account mentioned above. You can also use this as the Storage Credential for other External Locations if you like.

Search for Access Connector for Azure Databricks

Name you Access Connector and ensure it’s in the same Region as your Workspaces and Storage Account that you’ll use for the Unity Metastore.

Click Identity then Azure role assignments.

You don’t have to do this at the Resource Group Scope, it’s just what I used for this example. Most likely you’d want it at the Storage Account Scope.

Finally make sure to copy the Access Connector Resource ID

Enabling Unity:

We start off with a fresh new Workspace with no Unity.

Next, go to the dropdown menu in the upper right hand corner and click Manage Account.

As discussed in Prerequisite #1, you must either be your AD Global Administrator or have the Account admin role granted to you.

Once in, click either Create a metastore or the Data icon.

This will list all your metastores, if you have any yet. Remember multiple Workspaces can share the same metastore (if in the same Region), but a Workspace can only be assigned to one metastore. Click Create metastore to create a new one.

Again, ensure you are picking the correct Region. Enter in the <container_name>@<storage_account_name>.dfs.core.windows.net/<path> you wish to use for your metastore as well as the Access Connector Resource ID you copied earlier. Click Create.

Finally search for your Workspace and assign it to your newly created metastore.

Returning to your Catalog viewer in your Workspace you’ll now notice the name of the metastore at the top and two default new catalogs (main & systems)

Wrap Up:

High Level Overview of our steps

Congratulations, you have enabled Unity for your Workspace! You are now on your way to better Datalake management and set for the future. If you’re interested in how to migrate existing External Hive tables to Unity you can check out my post Migrate External Hive Streaming tables to Unity in an Azure Workspace.