Azure Databricks deployment’s Issues
Resource Groups
Databricks has some provision’s restrictions that sometimes throw the cloud team off if they are not prepare for them. In this series, I’ll go through them in details and record a demo video to explain it if you prefer to videos.
What are we deploying?
First things first, when deploying Databricks you deploy a Databricks workspace. A workspace is an Azure resource of type Microsoft.Databricks/workspaces. This is the only resource being deployed in the resource group that you choose to contain the workspace. Which brings us to the second point…
Managed Resource Group
Every Databricks workspace has a resource group managed by it and contains the resources that the workspace needs. Initially the group will contain
- Storage Account (that’s your dbfs)
- Virtual Network (if choose not to have vnet integration)
- Network Security Group (NSG) (if choose not to have vnet integration)
The group is locked by a read-only lock by the Databricks service principal which you can’t delete yourself even if you are the owner of the subscription.
The name of the resource group by default will be
databricks-rg-<workspace Name>-<random string>
You can see which resource group is managed by your workspace when you navigate to the workspace overview blade
Can I have control over the managed group name?
Yes! not from the portal but if you altered the ARM template, you choose the name you want for the managed resource group. This snippet shows that. A complete template is added at the end of the article as a reference
Can I combine multiple workspaces in the same managed Group?
No, one workspace, one managed resource group
Can I pre-create the resource group before creating the workspace?
Unfortunately no, the managed resource group has to be created by the workspace
How about tags? I must apply tags for my governance & cost management
Every time you create a tag on the workspace (during the creation in the ARM template or after the creation) it will be applied on the managed resource group as well. Databricks application keeps the sync.
When deleting a workspace, what happens to the managed resource group?
The resource group will be deleted by the service. It will take few minutes though to see them removed.
A whiteboard & demo video for this article is listed below.
Resources:
ARM Template for deploying Datbricks workspace with managed group parameter
https://github.com/mosharafMS/ARM-Templates/tree/master/Databricks/noVnetInjection
