Configure Azure Firewall with AKS egress Traffic using Terraform

In this post, I want to guide you on how you can configure the Azure firewall as a security check for all your outbound traffic. This configuration will make sure the remote services that your pod wants to interact with. By default, Once you deploy your pod in the cluster it can communicate to the external world.

Egress Network Traffic Flow [It Can be Multiple Cluster as well]

By seeing this diagram, you see the Pods inside won’t be able to connect to the internet until and unless you’re allowed from Azure Firewall policy. To achieve this security layer we need to configure the below services as followed.

1. Create a Spoke virtual network and a subnet [where Firewall needs to be configured].
2. Peering the Spoke virtual network with the Hub virtual network [where AKS will be hosted/it can be managed vnet for public cluster].
3. Create a Public IP and Azure Firewall service.
4. Create a route table and associate that with the hub subnet.
5. Configure network rules and application rules in Azure firewall policies.

Let's get Started with Terraform Code:

Assuming you have an AKS cluster running, In this Post we will be focusing on the egress traffic configuration part.

Reference to get the existing resource Group.

data "azurerm_resource_group" "rg-name" {
  name = var.resource_group_name
}

Create a Spoke Vnet and Subnet. Make sure there shouldn’t be any IP range conflict with the other virtual network address space.

resource "azurerm_virtual_network" "spoke-vnet" {
  address_space       = var.hub_vnet_address_space
  location            = data.azurerm_resource_group.rg-name.location
  name                = var.hub_vnet_name
  resource_group_name = data.azurerm_resource_group.rg-name.name
}
resource "azurerm_subnet" "firewall-subnet" {
  address_prefixes     = var.hub_subnet_address_space
  name                 = var.hub_subnet_name
  resource_group_name  = data.azurerm_resource_group.rg-name.name
  virtual_network_name = azurerm_virtual_network.hub-vnet.name
}

Create a Public IP and create an Azure Firewall Service. It’s recommended to create a static public IP and associate that with Firewall.

resource "azurerm_public_ip" "firewall-ip" {
  name                = "${var.firewall_name}-pip"
  location            = data.azurerm_resource_group.rg-name.location
  resource_group_name = data.azurerm_resource_group.rg-name.name
  allocation_method   = "Static"
  sku                 = "Standard"
}
resource "azurerm_firewall" "firewall" {
  name                = var.firewall_name
  location            = data.azurerm_resource_group.rg-name.location
  resource_group_name = data.azurerm_resource_group.rg-name.name
  sku_name            = "AZFW_VNet"
  sku_tier            = "Standard"
  ip_configuration {
    name                 = "configuration"
    subnet_id            = azurerm_subnet.firewall-subnet.id
    public_ip_address_id = azurerm_public_ip.firewall-ip.id
  }
}

Create a routing table and associate it with the spoke subnet. In the case of a public AKS cluster, you can add the managed subnet as data_source and you can associate it.

resource "azurerm_route_table" "aks-route-table" {
  depends_on                    = [azurerm_subnet.firewall-subnet]
  name                          = "aks-route-table"
  location                      = data.azurerm_resource_group.rg-name.location
  resource_group_name           = data.azurerm_resource_group.rg-name.name
  disable_bgp_route_propagation = false
  route {
    name                   = "aks-route"
    address_prefix         = "0.0.0.0/0"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = azurerm_firewall.firewall.ip_configuration[0].private_ip_address
  }
}
resource "azurerm_subnet_route_table_association" "aks-route-table" {
  subnet_id      = azurerm_subnet.aks-subnet.id
  route_table_id = azurerm_route_table.aks-route-table.id
}

Create network rules and application rules for Firewall Service. By this, here we have allowed some service tags along with some mandatory firewall rules mentioned by Microsoft. Here is the official list. Service Tags can be allowed for a specific region or globally as well. Here you can get the list here.

resource "azurerm_firewall_network_rule_collection" "aksfwnr" {
  name                = "aksfwnr"
  azure_firewall_name = azurerm_firewall.firewall.name
  resource_group_name = data.azurerm_resource_group.rg-name.name
  priority            = 101
  action              = "Allow"

  rule {
    name              = "time"
    source_addresses  = ["*"]
    destination_ports = ["123"]
    destination_addresses = ["*"]
    protocols = [
      "UDP"
    ]
  }
  rule {
    name                  = "apiudp"
    source_addresses      = ["*"]
    destination_ports     = ["9000"]
    destination_addresses = ["AzureCloud.${data.azurerm_resource_group.rg-name.location}"]
    protocols = [
      "TCP"
    ]
  }
  rule {
    name                  = "apitcp"
    source_addresses      = ["*"]
    destination_ports     = ["1194"]
    destination_addresses = ["AzureCloud.${data.azurerm_resource_group.rg-name.location}"]
    protocols = [
      "UDP"
    ]
  }
  rule {
    name                  = "dns"
    source_addresses      = ["*"]
    destination_ports     = ["53"]
    destination_addresses = ["*"]
    protocols = [
      "Any"
    ]
  }
}

resource "azurerm_firewall_network_rule_collection" "servicetags" {
  name                = "servicetags"
  azure_firewall_name = azurerm_firewall.firewall.name
  resource_group_name = data.azurerm_resource_group.rg-name.name
  priority            = 200
  action              = "Allow"

  rule {
    name              = "allow service tags"
    source_addresses  = ["*"]
    destination_ports = ["*"]
    destination_addresses = [
      "AzureContainerRegistry.${data.azurerm_resource_group.rg-name.location}",
      "MicrosoftContainerRegistry.${data.azurerm_resource_group.rg-name.location}",
      "AzureActiveDirectory",
      "AzureMonitor",
      "AzureWebPubSub",
      "Storage",
      "StorageSyncService"
    ]
    protocols = [
      "Any"
    ]
  }
}

You can Configure Application rules like below. As per best practices, it’s mandatory to add OS updates.

resource "azurerm_firewall_application_rule_collection" "osupdates" {
  name                = "osupdates"
  azure_firewall_name = azurerm_firewall.firewall.name
  resource_group_name = data.azurerm_resource_group.rg-name.name
  priority            = 102
  action              = "Allow"

  rule {
    name             = "allow network"
    source_addresses = ["*"]
    target_fqdns = [
      "download.opensuse.org",
      "security.ubuntu.com",
      "packages.microsoft.com",
      "azure.archive.ubuntu.com",
      "changelogs.ubuntu.com",
      "snapcraft.io",
      "api.snapcraft.io",
      "motd.ubuntu.com"
    ]
    protocol {
      port = "443"
      type = "https"
    }
    protocol {
      port = "80"
      type = "http"
    }
  }
}

Once the above configuration is applied via Terraform, To Perform a test, You can deploy an application which will start a test application by pulling the images from the docker hub or some other repos.

Or if you have some application which needs to interact with some other Azure services like Azure KeyVault/Cosmos DB which is not allowed as of now. The expected behaviour will be failing to connect to the service since it’s allowed.

In this post, we will be deploying a sample application which will try to pull an image from the docker hub. The below sample deployment file will create a deployment for the nginx application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Post-deployment, as expected it’s failing to pull images from the docker registry.

Pods are failing to pull images from Image Registry

Pod events where its showing failed to pull images

To fix this now we will allow the egress traffic via application rules in Azure firewall. To do that, we can add the below code in Terraform.

resource "azurerm_firewall_application_rule_collection" "dockerhub" {
  name                = "dockerhub"
  azure_firewall_name = azurerm_firewall.firewall.name
  resource_group_name = data.azurerm_resource_group.rg-name.name
  priority            = 200
  action              = "Allow"

  rule {
    name             = "allow network"
    source_addresses = ["*"]
    target_fqdns = [
      "*auth.docker.io",
      "*cloudflare.docker.io",
      "*cloudflare.docker.com",
      "*registry-1.docker.io"
    ]
    protocol {
      port = "443"
      type = "Https"
    }
    protocol {
      port = "80"
      type = "Http"
    }
  }
}

Post apply the changes, pods should be able to pull images from the mentioned image registry.

Pod Status

As you can see using the firewall rules we can control the egress traffic, Hope this helps in your environment.