Kubernetes Security — Benefits of Cilium clusterwide network policies

Photo by Alina Grubnyak on Unsplash

I wrote about Cilium network policies in a previous story.

Kubernetes Security — Control pod to pod communications with Cilium network policies

Today I will quickly highlight one advantage of Cilium clusterwide network policies, that is, network policies that are not scoped to a particular namespace but global to a cluster.

Setup from the previous story

In the previous story we created a local cluster with Kind, installed Cilium with policy enforcement enabled, and installed a couple of CiliumNetworkPolicy resources to whitelist the necessary traffic for various pods (including Hubble Relay and Hubble UI).

All the setup we did is summarized in the script below:

# create a local cluster
kind create cluster --image kindest/node:v1.23.3 \
  --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true        # do not install kindnet
  kubeProxyMode: none            # do not run kube-proxy
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
# install cilium
helm upgrade --install --namespace kube-system \
  --wait --timeout 15m --atomic \
  --repo https://helm.cilium.io cilium cilium \
  --values - <<EOF
kubeProxyReplacement: strict
k8sServiceHost: kind-external-load-balancer
k8sServicePort: 6443
policyEnforcementMode: always    # enforce network policies
hostServices:
  enabled: true
externalIPs:
  enabled: true
nodePort:
  enabled: true
hostPort:
  enabled: true
image:
  pullPolicy: IfNotPresent
ipam:
  mode: kubernetes
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
EOF
# core-dns network policy
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: core-dns
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: coredns
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: hubble-ui
  egress:
  - toEntities:
    - kube-apiserver
    - world
EOF
# local-path-provisioner network policy
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: local-path-provisioner
  namespace: local-path-storage
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: local-path-provisioner-service-account
  egress:
  - toEntities:
    - kube-apiserver
EOF
# hubble-relay network policy
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: hubble-relay
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: hubble-relay
  ingress:
  - fromEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: hubble-ui
  egress:
  - toEntities:
    - host
    - remote-node
EOF
# hubble-ui network policy
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: hubble-ui
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: hubble-ui
  egress:
  - toEntities:
    - kube-apiserver
  - toEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: hubble-relay
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: coredns
EOF

Use of Cilium clusterwide network policy

In the setup above, the network policy allowing ingress to core-dns pods is quickly going to become hard to manage.

Lots of workloads in a cluster need to talk with the internal DNS service and whitelisting every one of them can be a time consuming, repetitive, and error prone task.

To simplify that, we can take advantage of CiliumClusterwideNetworkPolicy to allow ingress from all pods to the core-dns pods. Eventually we can allow egress from all pods to the core-dns pods too.

# allow ingress from all pods to the core-dns pods
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: core-dns-ingress
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: coredns
      k8s:io.kubernetes.pod.namespace: kube-system
  ingress:
  - fromEndpoints:
    - {}
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
EOF
# allow egress from all pods to the core-dns pods
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: core-dns-egress
spec:
  endpointSelector: {}
  egress:
  - toEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: coredns
        k8s:io.kubernetes.pod.namespace: kube-system
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
EOF

With the cluster network policies above in place, we can simplify the core-dns and hubble-ui network policies, removing ingress and egress rules related to internal DNS service:

# no need to whitelist hubble-ui ingress anymore
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: core-dns
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: coredns
  egress:
  - toEntities:
    - kube-apiserver
    - world
EOF
# no need to whitelist core-dns egress anymore
kubectl apply -f - <<EOF
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: hubble-ui
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      io.cilium.k8s.policy.serviceaccount: hubble-ui
  egress:
  - toEntities:
    - kube-apiserver
  - toEndpoints:
    - matchLabels:
        io.cilium.k8s.policy.serviceaccount: hubble-relay
EOF

Wrapping it up

Thanks to CiliumClusterwideNetworkPolicy, allowing access to core services in the cluster can be expressed with a single resource.

This is to be used with precautions but comes very handy for legitimate use cases as it greatly simplifies the configuration.