Accessing Azure Kubernetes Service Private cluster services via Azure Private Link Service and Private Endpoints in Azure Data Factory and other PaaS Services in managed VNETs
Co-authored with Pankaj Tiwari
Background
Recently we had an interesting challenge to access a Kubernetes App running inside a Private AKS cluster from Azure Managed Service(Azure Data Factory, Azure Synapse, etc.) that runs in its own managed VNET via an integration runtime environment.
We as end users don't have any control over this managed VNET. Thus accessing services/APIs which are provisioned inside another VNET or on-premise setup is not possible since peering ofthe VNETs ( managed and our Private Kubernetes VNET ) cant be done. The managed VNET is abstracted under the hood and is not visible to us on the Azure Console.
It is also preferable and secure to maintain traffic navigation within Azure itself for inter-service communications, instead of traffic going outside of Azure and reaching services back again.
This article will be a walkthrough on how to establish connectivity between a managed VNET and an unmanaged VNET using Azure Private Endpoints and Azure Private Link Services.
In our case, we are running an App behind a Kubernetes service inside a Private AKS Cluster in a VNET and is exposed on an Azure internal load balancer.
We want to access this service from ADF(Azure Data Factory), which runs in a Managed Integration Runtime. The same tutorial can be replicated for other Azure services as well.
Setting it up
- Create a Private AKS Cluster on Azure Portal (https://learn.microsoft.com/en-us/azure/aks/private-clusters)
- Create an ADF workspace.
Kindly ensure that you have enabled Managed Virtual Network on the default AutoResolveIntegerationRuntime
3. Deploy an App on the Kubernetes Cluster.
Code: https://github.com/pkt1583/private-link-service-demo
Add the following annotations to the service.
These annotations help us to create a Private Link Service(PLS) resource automatically in Azure.
A private link service essentially helps services in one VNET to be accessible in another VNET privately and without traffic going to the public internet and stays within the Azure backbone.
Azure Private Link service is the reference to your own service that is powered by Azure Private Link. Your service that is running behind Azure Standard Load Balancer can be enabled for Private Link access so that consumers of your service can access it privately from their own VNets. Your customers can create a private endpoint inside their virtual network and map it to this service. This article explains concepts related to the service provider side.
4. The Kubernetes app deployed will be internal and not be available outside the VNET.
5. Go to Private Link Service(PLS) on Azure Portal and we should see PLS created under the Private Link Services section.
5. Now go to Azure Data Factory in the Azure Portal.
Launch ADF Studio and Add a Web Component to the Pipeline.
6. Navigate to Manage Section > Managed Private Endpoints.
7. Create a New Managed Private Endpoint and select the Private Link Service option.
The Full Qualified domain name can have any value.
8. We will see Provisioning State pending for a few minutes and ultimately approved.
9. Now go to Private Endpoints and we will see the Connection State as pending.
10. Click on Approve and it will change to approved.
11. Now let's go back to the Azure Data Factory workspace and to the Web component of the Pipeline add the URL that you supplied in the FQDN.
12. Execute the Pipeline and we will see the connection working successfully.
Conclusion
Thus we could connect to a service/app running inside a VNET inside a Private cluster from another VNET/managed VNET/on-premise system in any of the Azure PaaS Services.
