Setting Up Multi-Cloud (On-premises) Connectivity with Databricks

Introduction

This is a high-level guide on how to establish connectivity from your Databricks workspace on GCP to your other cloud / on-premises network. For the example detailed here, we will be setting up connectivity from Databricks on GCP to an Azure environment. A similar methodology would apply for connection to AWS or on-premises.

This guide is based on this blog post, with modifications implemented to support connectivity to and from Databricks. This guide can be leveraged for connecting Databricks on GCP to various data sources hosted in external environments (Azure, AWS, on-premises), either via direct private IP connections, or via hostname lookups through Cloud DNS.

Checklist

• [Azure] Create the following services: 1 x VNET, 1 x Default Subnet, 1 x Gateway Subnet, 1 x NSG, 2 x Public IPs, 1 x Virtual Network Gateway
• [GCP] Create a Databricks workspace
• [GCP] Create the following services: 1 x External IP, 1 x VPN Gateway
• [GCP] Define forwarding rules for VPN traffic
• [GCP] Create VPN Tunnel
• [Azure] Create Local Gateway
• [Azure] Create VPN Connection
• [GCP] Adjust Databricks Network Routes
• [GCP] Adjust Databricks VPC Firewall Rules
• [Databricks] Test connectivity via private IP calls
• [GCP] Setup DNS zone
• [Databricks] Test connectivity via hostname

Section 1: Ensure you have an Azure environment that meets the prerequisites

Required Azure components

1. Virtual Network
2. Subnet(s) for services / VMs / etc. that you want to connect to
3. Subnet for Gateway
4. Network Security Group
5. Public IPs (for Virtual Network Gateway, for VM used for testing)
6. Virtual Network Gateway

For this example, we will be hosting our resources in a Resource Group named “multicloud_vpn_rg”. In this RG, we define a virtual network named “mc_vnet”, with an IPv4 address space of 10.1.0.0/16.

As we’ll be using Azure PowerShell in this example, the first step involves authentication (which we do interactively here).

Connect-AzAccount -UseDeviceAuthentication -subscription XYZXYZXYZ

We then create an Azure Virtual Network, with 2 subnetworks attached — a “default” one which we will use to host a Virtual Machine used for pinging tests, as well as a “GatewaySubnet” that will host a Virtual Gateway Network.

Subnet(s) for services / VMs / etc. that you want to connect to:

$defaultSubnet = New-AzVirtualNetworkSubnetConfig `
-Name “default” `
-AddressPrefix “10.1.1.0/24”

Subnet for Gateway:

$gatewaySubnet = New-AzVirtualNetworkSubnetConfig  `
-Name "GatewaySubnet" `
-AddressPrefix "10.1.2.0/24"

Create virtual network and subnets:

$vnet = New-AzVirtualNetwork  `
  -Name "mc_vnet" `
  -ResourceGroupName "multicloud_vpn_rg" `
  -Location "West Europe" `
  -AddressPrefix "10.1.0.0/16" `
  -Subnet $gatewaySubnet,$defaultSubnetSubnet     

We have finished creation of items #1, #2, and #3 in this section.

NSG

Once we have the VNet, we also create a Network Security Group that we will later use to restrict inbound & outbound traffic on our subnets.

$nsg = New-AzNetworkSecurityGroup `
-ResourceGroupName “multicloud_vpn_rg” `
-Location “West Europe” `
-Name “nsg-vm”

We have finished creation of item #4, and customisation of Azure traffic restriction is out of scope for this guide.

Public IPs

We also need to create two Public IP Addresses which will be used for our VPN gateway and for the VM we will create.

Create public IP address for VM:

$vmpip = New-AzPublicIpAddress `
-Name “vm-ip” `
-ResourceGroupName “multicloud_vpn_rg” `
-Location “West Europe” `
-AllocationMethod Dynamic

Create public IP address for NW gateway:

$ngwpip = New-AzPublicIpAddress `
-Name “ngw-ip” `
-ResourceGroupName “multicloud_vpn_rg” `
-Location “West Europe” `
-AllocationMethod Dynamic

We have finished creation of item #5 in this section.

Virtual Network Gateway

Next, we deploy our Virtual Network Gateway, which will be used to create a VPN tunnel to our Google environment. This will be deployed into the “GatewaySubnet” that we defined at the start, and it will leverage the “ngw-ip” Public IP address.

$vnet = Get-AzureRmVirtualNetwork -Name mc_vnet -ResourceGroupName multicloud_vpn_rg
$gatewaySubnetId = $vnet.Subnets[0].id
$ngwpip = Get-AzPublicIpAddress -Name ngw-ip -ResourceGroupName multicloud_vpn_rg

Create virtual network gateway:

$ngwipconfig = New-AzVirtualNetworkGatewayIpConfig `
  -Name "ngw-ipconfig" `
  -SubnetId $gatewaySubnetId `
  -PublicIpAddressId $ngwpip.Id

This could take quite some time:

$job = New-AzVirtualNetworkGateway -Name “vnet-gateway” `
-ResourceGroupName “multicloud_vpn_rg” `
-Location “West Europe” `
-IpConfigurations $ngwipconfig `
-GatewayType “Vpn” `
-VpnType “RouteBased” `
-GatewaySku “VpnGw1” `
-VpnGatewayGeneration “Generation1” `
$vnetgw = Get-AzVirtualNetworkGateway `
  -Name "vnet-gateway" `
  -ResourceGroupName "multicloud_vpn_rg"

We have finished creation of item #6 in this section.

Section 2: Setup Databricks in your GCP environment

If you don’t have an existing Databricks workspace in GCP, please follow the documentation to create a new workspace. Please make sure that you are sizing the networks of your workspace correctly.

For this example, we’ll be using a Private GKE cluster deployment, and using the following subnets:

• Primary IP range for GKE nodes: 10.3.0.0/16
• Secondary IP range for GKE nodes: 10.4.0.0/16
• Secondary IP range for GKE services: 10.5.0.0/20
• IP range for GKE master resources: 10.6.0.0/28

Once your workspace is provisioned, do a quick validation test and start a small cluster and make sure everything’s working.

Finding Databricks Managed VPC / Networks

You can find the corresponding Databricks Managed VPC for your Databricks workspace by looking at the URL of your workspace on the admin console (the Workspace ID), and matching this to a partial VPC ID. Databricks Managed VPCs follow this naming format: databricks-managed-<workspaceid>. From there, you can then also review all the subnetworks used by your Databricks instance.

Section 3: Setup required GCP components to ensure VPN connectivity to Azure

Required GCP components

1. External IP (for Virtual Network Gateway)
2. Virtual Network Gateway (and relevant forwarding rules)

External IP

We first create an external IP in our GCP environment to connect the Azure resource group to GCP via the VPN Gateway.

gcloud compute addresses create “ext-gw-ip” \
 — region “europe-west2”

We have finished creation of item #1 in this section.

Virtual Network Gateway

Now we need to establish a VPN connection between the two networks (Azure and GCP). A connection has typically two exits. In our case, one exit is on the Azure side (where we created the Azure Virtual Network Gateway) and the second exists on the GCP side (as we will be creating a VPN Gateway in this subsequent step).

To connect those two gateways, we need to link the gateway in Azure to the gateway in GCP. Each of the gateways is acting like an entry to its corresponding network and that’s why we need to define traffic rules coming from one gateway (respectively Azure virtual Network Gateway/VPN gateway), going through the second gateway (respectively VPN gateway/Azure Virtual Network Gateway) to the resources in the target network (respectively VPC/VNET).

If you need to remind yourself what the managed network is, please review the previous section.

Create cloud VPN:

gcloud compute target-vpn-gateways create “vpn-gw” \
 — network “databricks-managed-4508843396878646” \
 — region “europe-west2” \
 — project “fe-dev-sandbox”

Just having the VPN Gateway in place is not enough. We also need to allow for connectivity; traffic rules are defined using forwarding rules on the GCP side. Each forwarding rule references an IP address and one or more ports on which the VPC accepts traffic.

Create forwarding rule ESP:

gcloud compute forwarding-rules create "fr-gw-name-esp" \
 - ip-protocol ESP \
 - address "ext-gw-ip" \
 - target-vpn-gateway "vpn-gw" \
 - region "europe-west2" \
 - project "fe-dev-sandbox"

Creating forwarding rule UDP500:

gcloud compute forwarding-rules create "fr-gw-name-udp500" \
 - ip-protocol UDP \
 - ports 500 \
 - address "ext-gw-ip" \
 - target-vpn-gateway "vpn-gw" \
 - region "europe-west2" \
 - project "fe-dev-sandbox"

Creating forwarding rule UDP4500:

gcloud compute forwarding-rules create "fr-gw-name-udp4500" \
 - ip-protocol UDP \
 - ports 4500 \
 - address "ext-gw-ip" \
 - target-vpn-gateway "vpn-gw" \
 - region "europe-west2" \
 - project "fe-dev-sandbox"

We have finished creation of item #2 in this section.

Section 4: Establish VPN Connection

We now need to setup the VPN connection on both sides:

• Azure side of VPN connection — Establish a connection from Azure virtual Network Gateway ($vnetgw) to Azure Local Gateway (Acting like remote GCP Gateway since it is linked to the External IP on GCP side)
• GCP side of VPN connection — Establish a connection from GCP VPN Gateway ($vpn-gw) to the Public IP Address on Azure side ($azpubip)

Required steps

1. [GCP] Create a connection from GCP VPN Tunnel
2. [Azure] Create Azure Local Network Gateway
3. [Azure] Create Azure Virtual Local Network Connection

GCP VPN Tunnel

First we create a connection in our GCP environment (VPN Tunnel) using the Public IP address of the Azure Virtual Network Gateway. As this example uses a route based VPN the traffic selector values need to be set at 0.0.0.0/0. A PSK (Pre Shared Key) needs to be supplied which will be the same key used when we configure a VPN Connection on the Azure side of the tunnel.

If you forgot your Azure Public IP, you can run the following command in your Azure environment:

Get peer public IP address of Azure gateway:

$azpubip = Get-AzPublicIpAddress `
-Name "ngw-ip" `
-ResourceGroupName "multicloud_vpn_rg"

When you’re ready to start, you can generate your PSK, then create the VPN tunnel in your GCP environment:

openssl rand -base64 24

Save this key. You will need it later on in this process.

Create VPN tunnel:

gcloud compute vpn-tunnels create "vpn-tunnel-to-azure" \
 - peer-address $azpubip.IpAddress \
 - local-traffic-selector "0.0.0.0/0" \
 - remote-traffic-selector "0.0.0.0/0" \
 - ike-version 2 \
 - shared-secret <<Pre-Shared Key>> \
 - target-vpn-gateway "vpn-gw" \
 - region "europe-west2" \
 - project "fe-dev-sandbox"

We have finished creation of item #1 in this section.

Azure Local Gateway

We now create an Azure local Gateway acting like a remote Gateway for GCP. For this, we will need:

• The GCP External IP
• The GCP VPC IP address ranges that you entered when setting up your Databricks workspace

Please review section 2 to retrieve infos about your Databricks managed VPC.

$azlocalgw = New-AzLocalNetworkGateway `
-Name "local-gateway" `
-ResourceGroupName "multicloud_vpn_rg" `
-Location "West Europe" `
-GatewayIpAddress $gcp_ipaddr `
-AddressPrefix @("10.3.0.0/16","10.4.0.0/16","10.5.0.0/20")

We have finished creation of item #2 in this section.

Azure VPN Connection

Next, we create a connection in our Azure environment (VPN connection) by associating the Azure Virtual Network Gateway with the Local Network Gateway. A PSK (Pre Shared Key) needs to be supplied which is the same key used for the GCP VPN Tunnel.

$azvpnconn = New-AzVirtualNetworkGatewayConnection `
-Name "vpn-connection" `
-ResourceGroupName "multicloud_vpn_rg" `
-VirtualNetworkGateway1 $vnetgw `
-LocalNetworkGateway2 $azlocalgw `
-Location "West Europe" `
-ConnectionType IPsec `
-SharedKey << Pre-Shared Key >> `
-ConnectionProtocol "IKEv2"

After a few minutes (~5 min), you should see both the Azure connection status change to “Connected”, as well as GCP VPN tunnel status change to “Established”

We have finished creation of item #3 in this section.

Section 5: Make adjustments to Databricks environment to leverage VPN connectivity

You will need to customise the Databricks environment (Mainly the Databricks Managed VPC) created automatically upon the Databricks workspace deployment to leverage VPN connectivity. For this, you will also need your Databricks Managed VPC ID.

Since this is an external connectivity, we need to make sure that is secure by defining how traffic should flow into our GCP network (routes) and then by allowing this traffic into our GCP network (Firewall rules).

To note, we will only address limited data exfiltration rules, and exclusively in our GCP environment where Databricks is hosted. For more details on Firewall setup with Databricks, please review dedicated documentation.

Required steps

1. [GCP] Setup Static Routes
2. [GCP] Setup Firewall Rules

Static Routes

Now that we have network resources on both ends (a VNET in Azure, and a VPC created and managed by Databricks in GCP), we need to route traffic on the GCP side by setting up routes for outgoing traffic to the Azure network.

gcloud compute routes create "route-to-azure" \
 - destination-range "10.1.0.0/16" \
 - next-hop-vpn-tunnel "vpn-tunnel-to-azure" \
 - network "databricks-managed-4508843396878646" \
 - next-hop-vpn-tunnel-region "europe-west2" \
 - project "fe-dev-sandbox"

We have finished the creation of item #1 in this section.

Firewall Rules

To enforce security, now that we defined the routes (ie directions) of the traffic (GCP → Azure), we will also need to specify firewall rules to allow this traffic out of the Databricks managed VPC (i.e. allow VPN traffic).

gcloud compute firewall-rules create "vpn-rule1" \
 - network "databricks-managed-4508843396878646" \
 - allow tcp,udp,icmp \
 - source-ranges "10.1.0.0/16"​​

We have finished the creation of item #2 in this section.

Section 6: Test connectivity between Databricks in GCP and an Azure VM

In this section, we quickly created a VM via the Azure console, with the settings below. We validated that we could RDP into the machine using my local computer.

Once we remotely connect into the Azure VM, we can start a command prompt and ping the internet just to make sure that it works ok.

We now start up a cluster in Databricks on GCP. We issue a ping command from within Databricks to the Azure VMs private IP address.

Section 7 [optional]: Setup Private DNS zone to be able to use hostnames to connect to Azure services

If we want to use Private DNS in order to call some of these services by a custom-defined name, rather than by direct IP addresses, we can create a Cloud DNS zone at the project level, and apply it to our managed Databricks network.

gcloud beta dns - project=fe-dev-sandbox managed-zones create dns-azurevm \
 - description="Translation of hostname to Azure VM private IP in multicloud setup" \
 - dns-name="azurevm.databricks.com." \
 - visibility="private" \
 - networks="databricks-managed-4508843396878646"

We can then define the record sets for this zone. In the example below, we define vm1.azurevm.databricks.com as the name for the private IP of our Azure VM 10.1.1.4.

gcloud dns - project=fe-dev-sandbox \
record-sets transaction start - zone=dns-azurevm
gcloud dns - project=fe-dev-sandbox \
record-sets transaction add "10.1.1.4" \
 - name=vm1.azurevm.databricks.com. \
 - ttl=300 \
 - type=A \
 - zone=dns-azurevm
gcloud dns - project=fe-dev-sandbox \
record-sets transaction execute \
 - zone=dns-azurevm

Once we have done the steps above, we can go back to Databricks in GCP, and issue a ping command, this time using the newly defined name for our VM.

Outro

We have successfully set up Multi-Cloud VPN connectivity between GCP and Azure, and are able to send requests from Databricks on GCP to a VM within our Azure environment using private traffic. We have also set up a DNS zone to facilitate how we do this within our Databricks on GCP environment.

If you want to learn more about Databricks or Google Cloud, please reach out via the linked contact pages.

The views expressed are those of the authors and don’t necessarily reflect those of Google or Databricks

Please clap for this article if you enjoyed reading it. For more google cloud-based data science, data engineering, and AI/ML follow me on LinkedIn.

Big thanks to Silviu Tofan and Marwa Krouma from Databricks and Kristina Florea and Peter Greiff from Google Cloud for supporting this article and Databricks on GCP.