Setting up azure firewall for analysing outgoing traffic in AKS

Every now and then we get the question on how to lock down ingoing to and outgoing traffic from a kubernetes cluster in azure. One option that can be set up relatively easy but is not documented in detail is using the Azure Firewall (https://azure.microsoft.com/en-us/services/azure-firewall/).

My personal recommendation on this scenario is to use the firewall to diagnose and watch the network dependencies of your applications — which is why I am also documenting the services that are currently needed for AKS to run. If you turn on the rules to block outgoing traffic, you risk that your cluster breaks if the engineering team brings in additional required network dependencies. Be careful on production environments with this.

The end result will look like this and requires some steps to configure the vnet, subnets, routetable, firewall rules and azure kubernetes services which are described belown and can be adapted to any kubernetes installation on azure:

First we will set up the vnet — I prefer using azure cli in shell.azure.com over powershell but you can easy achieve the same using terraform or arm. If you have preferences on the naming conventions please adjust the variables below. In most companies the vnet is provided by the networking team, so we should assume that the network configuration will not be done by the team which is maintaining the AKS cluster.

Lets define some variables first:

SUBSCRIPTION_ID="" # here enter your subscription id
KUBE_GROUP="kubes_fw_knet" # here enter the resources group name of your AKS cluster
KUBE_NAME="dzkubekube" # here enter the name of your kubernetes resource
LOCATION="westeurope" # here enter the datacenter location
KUBE_VNET_NAME="knets" # here enter the name of your vnet
KUBE_FW_SUBNET_NAME="AzureFirewallSubnet" # this you cannot change
KUBE_ING_SUBNET_NAME="ing-4-subnet" # here enter the name of your ingress subnet
KUBE_AGENT_SUBNET_NAME="aks-5-subnet" # here enter the name of your AKS subnet
FW_NAME="dzkubenetfw" # here enter the name of your azure firewall resource
FW_IP_NAME="azureFirewalls-ip" # here enter the name of your public ip resource for the firewall
KUBE_VERSION="1.11.5" # here enter the kubernetes version of your AKS
SERVICE_PRINCIPAL_ID= # here enter the service principal of your AKS
SERVICE_PRINCIPAL_SECRET= # here enter the service principal secret
  1. Select subscription, create the resource group and the vnet
az account set --subscription $SUBSCRIPTION_ID
az group create -n $KUBE_GROUP -l $LOCATION
az network vnet create -g $KUBE_GROUP -n $KUBE_VNET_NAME

2. Assign permissions on vnet for your service principal — usually “virtual machine contributor” is enough

az role assignment create --role "Virtual Machine Contributor" --assignee $SERVICE_PRINCIPAL_ID -g $KUBE_GROUP

3. Create subnets for the firewall, ingress and AKS

az network vnet subnet create -g $KUBE_GROUP --vnet-name $KUBE_VNET_NAME -n $KUBE_FW_SUBNET_NAME --address-prefix 10.0.3.0/24
az network vnet subnet create -g $KUBE_GROUP --vnet-name $KUBE_VNET_NAME -n $KUBE_ING_SUBNET_NAME --address-prefix 10.0.4.0/24
az network vnet subnet create -g $KUBE_GROUP --vnet-name $KUBE_VNET_NAME -n $KUBE_AGENT_SUBNET_NAME --address-prefix 10.0.5.0/24 --service-endpoints Microsoft.Sql Microsoft.AzureCosmosDB Microsoft.KeyVault Microsoft.Storage

As you might know there are two different options on how networking can be set up in AKS called “Basic networking” (using kubenet cni) and “Advanced Networking” (using azure cni). I am not going into detail how they differ — you can look it up here: https://docs.microsoft.com/en-us/azure/aks/concepts-network . For the usage of azure firewall in this scenario it does not matter since both options work but need to be configured differently, which is why I am documenting both options.

Basic networking requires you to modify the routetable that will be created by the AKS deployment, add another route and point it towards the internal ip of the azure firewall. If you want to use advanced networking skip this section and continue in the Advanced Networking section.

KUBE_AGENT_SUBNET_ID="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$KUBE_GROUP/providers/Microsoft.Network/virtualNetworks/$KUBE_VNET_NAME/subnets/$KUBE_AGENT_SUBNET_NAME"
az aks create --resource-group $KUBE_GROUP --name $KUBE_NAME --node-count 2 --network-plugin kubenet --vnet-subnet-id $KUBE_AGENT_SUBNET_ID --docker-bridge-address 172.17.0.1/16 --dns-service-ip 10.2.0.10 --service-cidr 10.2.0.0/24 --client-secret $SERVICE_PRINCIPAL_SECRET --service-principal $SERVICE_PRINCIPAL_ID --kubernetes-version $KUBE_VERSION --no-ssh-key

After the deployment went through you need go to the azure portal and create an instance of the azure firewall like this:

Make sure that the azure firewall deployment dialog uses the right names as specified in the variables.

After the deployment we create another route in the routetable and associate the route table to the subnet — which is required due to the known bug that is currently in AKS see here. If not already there you should install the azure firewall cli extension:

az extension add --name azure-firewall
FW_ROUTE_NAME="${FW_NAME}_fw_r"
FW_PUBLIC_IP=$(az network public-ip show -g $KUBE_GROUP -n $FW_IP_NAME --query ipAddress)
FW_PRIVATE_IP="10.0.3.4"
AKS_MC_RG=$(az group list --query "[?starts_with(name, 'MC_${KUBE_GROUP}')].name | [0]" --output tsv)
ROUTE_TABLE_ID=$(az network route-table list -g ${AKS_MC_RG} --query "[].id | [0]" -o tsv)
ROUTE_TABLE_NAME=$(az network route-table list -g ${AKS_MC_RG} --query "[].name | [0]" -o tsv)
AKS_NODE_NSG=$(az network nsg list -g ${AKS_MC_RG} --query "[].id | [0]" -o tsv)
az network vnet subnet update --resource-group $KUBE_GROUP --route-table $ROUTE_TABLE_ID --network-security-group $AKS_NODE_NSG --ids $KUBE_AGENT_SUBNET_ID
az network route-table route create --resource-group $AKS_MC_RG --name $FW_ROUTE_NAME --route-table-name $ROUTE_TABLE_NAME --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FW_PRIVATE_IP --subscription $SUBSCRIPTION_ID

Advanced networking is a bit simpler but requires you to create the routetable first, create the route and then again associate it with the AKS subnet. You can do this before the cluster gets created.

KUBE_AGENT_SUBNET_ID="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$KUBE_GROUP/providers/Microsoft.Network/virtualNetworks/$KUBE_VNET_NAME/subnets/$KUBE_AGENT_SUBNET_NAME"
az aks create --resource-group $KUBE_GROUP --name $KUBE_NAME --node-count 2  --network-plugin azure --vnet-subnet-id $KUBE_AGENT_SUBNET_ID --docker-bridge-address 172.17.0.1/16 --dns-service-ip 10.2.0.10 --service-cidr 10.2.0.0/24 --client-secret $SERVICE_PRINCIPAL_SECRET --service-principal $SERVICE_PRINCIPAL_ID --kubernetes-version $KUBE_VERSION --no-ssh-key

The firewall needs to be created like above, after that is done we can create the routetable and the route.

FW_ROUTE_NAME="${FW_NAME}_fw_r"
FW_ROUTE_TABLE_NAME="${FW_NAME}_fw_rt"
FW_PUBLIC_IP=$(az network public-ip show -g $KUBE_GROUP -n $FW_IP_NAME --query ipAddress)
FW_PRIVATE_IP="10.0.3.4"
az network route-table create -g $KUBE_GROUP --name $FW_ROUTE_TABLE_NAME
az network vnet subnet update --resource-group $KUBE_GROUP --route-table $FW_ROUTE_TABLE_NAME --ids $KUBE_AGENT_SUBNET_ID
az network route-table route create --resource-group $KUBE_GROUP --name $FW_ROUTE_NAME --route-table-name $FW_ROUTE_TABLE_NAME --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FW_PRIVATE_IP --subscription $SUBSCRIPTION_ID

Now that the firewall works lets set up diagnostics and import the dashboard file as described here: https://docs.microsoft.com/en-us/azure/firewall/tutorial-diagnostics.

We can see that the firewall is working correctly and blocking ubuntu.com but not www.ubuntu.com

A network rule matches the traffic flow then no other application rule will be applied. For AKS to work you need to allow the following network traffic in the subnet:

az extension add --name azure-firewall
az network firewall network-rule create --firewall-name $FW_NAME --collection-name "aksnetwork" --destination-addresses "*"  --destination-ports 22 443 --name "allow network" --protocols "TCP" --resource-group $KUBE_GROUP --source-addresses "*" --action "Allow" --description "aks network rule" --priority 100

Currently AKS needs the following outgoing network dependencies:

  • `*.<region>.azmk8s.io` (eg. `*.westeurope.azmk8s.io`) — this is the dns that is running your masters
  • `*cloudflare.docker.io` — This is a CDN endpoint for cached Container Images on Docker Hub.
  • `*registry-1.docker.io` — This is Docker Hub’s registry — a couple of images are there so we need to allow this.
az network firewall application-rule create  --firewall-name $FW_NAME --collection-name "aksbasics" --name "allow network" --protocols http=80 https=443 --source-addresses "*" --resource-group $KUBE_GROUP --action "Allow" --target-fqdns "*.azmk8s.io" "*auth.docker.io" "*cloudflare.docker.io" "*cloudflare.docker.com" "*registry-1.docker.io" --priority 100

Optionally you might want to allow the following traffic:

  • - `*.ubuntu.com, download.opensuse.org` — This is needed for security patches and updates — if the customer wants them to be applied automatically
  • - `*azurecr.io` — storing your images in azure container registry
  • - `*blob.core.windows.net` — the store behind acr
  • - `login.microsoftonline.com` — for azure aad login
  • - `dc.services.visualstudio.com` — application insights
  • - `*.opinsights.azure.com` — azure monitor
az network firewall application-rule create  --firewall-name $FW_NAME --collection-name "akstools" --name "allow network" --protocols http=80 https=443 --source-addresses "*" --resource-group $KUBE_GROUP --action "Allow" --target-fqdns "download.opensuse.org" "login.microsoftonline.com" "*.ubuntu.com" "*azurecr.io" "*blob.core.windows.net" "dc.services.visualstudio.com" "*.opinsights.azure.com" --priority 101
az network firewall application-rule create  --firewall-name $FW_NAME --collection-name "osupdates" --name "allow network" --protocols http=80 https=443 --source-addresses "*" --resource-group $KUBE_GROUP --action "Allow" --target-fqdns "download.opensuse.org" "*.ubuntu.com" --priority 102

Now all outgoing traffic will be filtered and you can check that by launching a pod and see if you can curl the outside internet.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: centos
spec:
containers:
- name: centoss
image: centos
ports:
- containerPort: 80
command:
- sleep
- "3600"
EOF

Now log inside and verify that we cannot curl to an external url that is not allowed while www.ubuntu.com is working fine.

Azure firewall working as expected, allowing www.ubuntu.com but blocking everything else.

One more thing might interest you: “How to expose a kubernetes service through the azure firewall?” If you expose a service through the normal LoadBalancer with a public ip, it will not be accessible because the traffic that has not been routed through the azure firewall will be dropped on the way out. Therefore you need to create your service with a fixed internal ip, internal LoadBalancer and route the traffic through the azure firewall both for outgoing and incoming traffic. If you only want to use internal load balancers, then you do not need to do this.

kubectl run nginx --image=nginx --port=80
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: nginx-internal
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "ing-4-subnet"
spec:
type: LoadBalancer
loadBalancerIP: 10.0.4.4
ports:
- port: 80
selector:
run: nginx
EOF

Now we retrieve the internal load balancer ip and register it in the azure firewall as a Dnat rule.

SERVICE_IP=$(kubectl get svc nginx-internal --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
az network firewall nat-rule create  --firewall-name $FW_NAME --collection-name "inboundlbrules" --name "allow inbound load balancers" --protocols "TCP" --source-addresses "*" --resource-group $KUBE_GROUP --action "Dnat"  --destination-addresses $FW_PUBLIC_IP --destination-ports 80 --translated-address $SERVICE_IP --translated-port "80"  --priority 101

Now you can access the internal service by going to the public ip of your azure firewall on port 80

open http://$FW_PUBLIC_IP:80
Our internal service is no reachable through the azure firewall public ip on port 80

THE END ;-)