Kubernetes Networking with Cilium CNI and OKE on Oracle Cloud

Published in

Oracle Developers

10 min readJul 21, 2022

Until recently, when you used OKE for your Kubernetes workload, the only networking option for your pod networking was your venerable and friendly neighborhood CNI, flannel. Flannel is comparatively lean and simple. It only handles your pod networking and isn’t interested in doing anything else.

However, Kubernetes use cases have exploded in complexity, variety, and other *ties. As a result, we’ve witnessed a Cambrian explosion of CNI development. There are now at least 25 of them, and recently we joined the party by releasing our VCN-native pod networking. It’s quite a mouthful, so for brevity I’ll call it “NPN.” NPN makes it possible for Kubernetes pods to have an IP address from a subnet in the VCN. Imagine your Kubernetes pods being in the same network as your VMs or your baremetal nodes or other workloads. That’s NPN.

The possibilities offered by the NPN CNI are enormous and we’re already working to add support for it in terraform-oci-oke. As I’ve not fully grasped the full magnitude of this yet, I’ll refrain from pontificating on it so you can read my colleagues Ajay and Greg’s blog announcement or the documentation for more details in the meantime.

I mentioned earlier that flannel is lean and simple and only concerns itself with the networking of pods. Well, mostly. If you are managing your own Kubernetes clusters on OCI and have additional needs e.g. Network Policy, chances are you might pick another CNI from the list e.g. ClusterAPI for OCI installs Calico by default. As OKE is a managed service, you cannot just replace flannel with one of your choice. Instead, you can deploy another CNI that 1) co-exists well with flannel and 2) offers Network Policy as a feature. Using Calico for Network Policy is well-documented with OKE. It has been used for quite a while now and it’s an option on the Terraform module too.

But for today we’ll explore a new CNI: Cilium. Why Cilium, you ask? For one, my team mate Sherwood Zern has been talking about it for quite a while. In fact, he hasn’t stopped talking about it. For another, I want to explore some its features, in particular eBPF.

In this article, we’ll look at the following:

How to deploy Cilium as a CNI on OKE
How to implement Kubernetes network policies using Cilium
Observability using Hubble, Prometheus and Grafana

Installing Cilium as a CNI on OKE

Provision an OKE cluster using whatever means you find comfortable: the OCI Console Quick Create, Terraform OKE Module or the command line. If you use the OCI Console, ensure you select Custom Create > Flannel Overlay:

At this point, if you are just curious and want to take it for a spin, you can assign a public IP address to the API endpoint. If you still want to use a private endpoint, you’ll need to be able to reach your API server at some point (at least for the purpose of this article). If that’s the case, review your cluster access options or you can use Terraform OKE Module and enable the operator host with instance_principal.

For node pools ensure you have at least 1 node pool with at least 2 worker nodes. That’s because by default Cilium deploys 2 replicas. If you are using Terraform OKE Module, you may use the sample code in the gist below:

https://gist.githubusercontent.com/hyder/030c2c66bd07b1cd65dd27e8009844b1/raw/4f0a38e61f91d95ac9ff1a89eead8478ad4815eb/cluster.tf

Fill in the values in your terraform.tfvars, run terraform init and apply and wait for the cluster to be created. I’ll assume from here you are using Terraform OKE Module and you have enabled the operator. ssh to the operator and check you are able to reach the API server. Note the handy short-hand aliases (k, h) for kubectl on the operator host:

[opc@dev-operator ~]$ k get nodes
NAME          STATUS   ROLES   AGE   VERSION
10.0.108.37   Ready    node    54m   v1.23.4
10.0.93.111   Ready    node    54m   v1.23.4

Add the cilium helm repo and generate the manifest locally for editing:

helm repo add cilium https://helm.cilium.io/
helm show values cilium/cilium > cilium.yaml

Modify the cilium.yaml and change the default to the following values:

containerRuntime:
  integration: crio
hubble:
  tls:
    enabled: false
hubble:
  relay:
    enabled: true
hubble:
  ui:
    enabled: true
ipam:
  mode: "kubernetes"
clustermesh:
  useAPIServer: true

Install cilium:

helm install cilium cilium/cilium --namespace=kube-system -f cilium.yaml

Download the cilium cli:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

And check the status. At this point, you’ll get some errors because some pods are not yet managed by cilium, probably because they started before Cilium itself was running:

[opc@dev-operator ~]$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         1 errors
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/
DaemonSet         cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment        hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Deployment        hubble-relay       Desired: 1, Unavailable: 1/1
Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Containers:       cilium             Running: 2
                  hubble-ui          Running: 1
                  hubble-relay       Running: 1
                  cilium-operator    Running: 2
Cluster Pods:     2/5 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.0@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade: 2
                  hubble-ui          quay.io/cilium/hubble-ui-backend:v0.9.0@sha256:000df6b76719f607a9edefb9af94dfd1811a6f1b6a8a9c537cba90bf12df474b: 1
                  hubble-ui          quay.io/cilium/hubble-ui:v0.9.0@sha256:0ef04e9a29212925da6bdfd0ba5b581765e41a01f1cc30563cef9b30b457fea0: 1
                  hubble-relay       quay.io/cilium/hubble-relay:v1.12.0@sha256:ca8033ea8a3112d838f958862fa76c8d895e3c8d0f5590de849b91745af5ac4d: 1
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.0@sha256:bb2a42eda766e5d4a87ee8a5433f089db81b72dd04acf6b59fcbb445a95f9410: 2
Errors:           hubble-relay       hubble-relay    1 pods of Deployment hubble-relay are not ready

Notice how only 2/5 pods are managed by cilium. We need to ensure all of them are managed by cilium. Let’s hunt those that are not. Cilium very helpfully provides a script to help you identify them:

curl -sLO https://raw.githubusercontent.com/cilium/cilium/master/contrib/k8s/k8s-unmanaged.sh
chmod +x k8s-unmanaged.sh

You’ll get something like the output below:

[opc@dev-operator ~]$ ./k8s-unmanaged.sh
Skipping pods with host networking enabled...
error: the server doesn't have a resource type "cep"
kube-system/coredns-5695dcc5cf-44rm6
kube-system/coredns-5695dcc5cf-pwj9g
kube-system/hubble-relay-859488fdc9-h6d75
kube-system/hubble-ui-5cbf89dcff-7nqvj
kube-system/kube-dns-autoscaler-6d89b5d4c-x92wz

Delete the above pods and let them be recreated:

kubectl -n kube-system delete pod <pod-name-1> <pod-name-2>

Check the status again and all lights should be green and all pods should now be managed by cilium:

cilium status

At this point, we can get remove the flannel daemonset since we don’t need it:

kubectl delete -n kube-system daemonset kube-flannel-ds

Let’s scale out the node pool and double it in size to 4. Our cilium status should still be all green. Yep, it is:

We can now run the connectivity test:

cilium connectivity test

and all our tests should pass:

Let’s now test the Hubble UI. If you created your cluster a public cluster and all the kubectl/helm were being done locally, you can just run the hubble UI as follows:

cilium hubble ui

The cilium cli will automatically set up the necessary port forwarding and it will also open a browser locally. If you are doing all your interactions through the operator, open a new terminal and ssh to it as follows to create a tunnel:

ssh -i ~/.ssh/id_rsa -J opc@<bastion-ip> opc@<operator-ip> -L 8080:localhost:8080 -L 8081:localhost:8081

Once you are logged in, get the name of the Hubble UI pod and use kubectl to do a port-forwarding to it:

POD_NAME=$(kubectl -n kube-system get pod --selector=k8s-app=hubble-ui  -o jsonpath='{.items[*].metadata.name}')
kubectl -n kube-system port-forward $POD_NAME 8081:8081

On your browser, access the Hubble UI on port 8081: http://localhost:8081. On a separate terminal, run the connectivity test again and select the cilium-test namespace in the Hubble UI:

Cilium is now fully operational in OKE.

Testing network policies

Let’s now test a simple network policy of denying all. Run a nginx Pod with labels app=web and expose it at port 80:

kubectl run web --image=nginx --labels="app=web" --expose --port=80

Run a temporary Pod and make a request to “web” Service:

kubectl run --rm -i -t --image=alpine test-$RANDOM -- sh
If you don't see a command prompt, try pressing enter.
/ # wget -qO- http://web
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

We get a response. In the Hubble UI, we can also see the traffic as being forwarded:

Let’s now create a network policy to deny all traffic:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-deny-all
spec:
  podSelector:
    matchLabels:
      app: web
  ingress: []

Save this to a web-deny-all.yaml and apply it:

kubectl apply -f web-deny-all.yamlnetworkpolicy.networking.k8s.io/web-deny-all created

In the temporary pod, run the command to access nginx again but setting a 2s timeout:

wget -qO- --timeout=2 http://web

This time, it times out. In the Hubble UI flow logs, we also see the traffic being dropped:

You can experiment with other network policies or with Cilium’s own network policy tutorials to get an idea of what’s possible when using Cilium for network policy.

Delete the network policies you’ve created so we can proceed with the 3rd part of this exercise e.g.

kubectl delete -f web-deny-all.yaml

Capturing network metrics

The last bit we want to do is understand how our network is behaving. First, we need to install Prometheus and Grafana. We’ll use kube-prometheus-stack.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm show values prometheus-community/kube-prometheus-stack > kube-prometheus-stack.yaml

Edit the kube-prometheus-stack.yaml and change the following:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false

And use helm to install the kube-prometheus-stack chart:

helm install kps --namespace monitoring prometheus-community/kube-prometheus-stack -f kube-prometheus-stack.yaml --create-namespace

Next, edit the cilium.yaml and change the following:

prometheus:
  enabled: true
prometheus:
  serviceMonitor:
    enabled: true
operator:
  prometheus:
    enabled: true
operator:
  prometheus:
    serviceMonitor:
      enabled: true
hubble:
  metrics:
    enabled:
    - dns
    - drop
    - tcp
    - flow
    - icmp
    - http
hubble:
  metrics:
    serviceMonitor:
      enabled: true
hubble:
  relay:
    prometheus:
      enabled: true
hubble:
  relay:
    prometheus:
      serviceMonitor:
        enabled: true

Upgrade the helm release with the new configuration that includes the metrics:

helm upgrade cilium cilium/cilium --namespace=kube-system -f cilium.yaml

Once the deployment is completed, do a port-forward to the Prometheus service:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 8080:9090

and on another terminal to the grafana service:

kubectl -n monitoring port-forward svc/kps-grafana 8081:80

To access the Prometheus UI so you can experiment with your Prometheus queries, open your browser on http://localhost:8080/ and to access Grafana, open your browser on http://localhost:8081. Login with the default Grafana username & password: admin/prom-operator. Then import the dashboards with the following IDs:

On Prometheus, you should be able to see the cilium metrics and targets:

Run cilium connectivity test again to generate some traffic:

cilium connectivity test

On the graph page for Prometheus, enter the following query and hit execute:

sum(rate(cilium_k8s_client_api_calls_total{k8s_app=”cilium”}[1m])) by (pod, method, return_code)

You should be able to see a graph like this:

On the Grafana dashboards, you’ll be able to find more interesting information. For example, the screenshot below shows part of the Hubble metrics dashboard

whereas the following one shows you part of the Cilium Agent metrics dashboard:

Although I can take a guess or look at the Cilium monitoring documentation, I actually don’t know what most of these metrics mean yet but I like my charts to be (mostly) populated and colourful. More importantly, they tell me the co-existence of flannel and Cilium on OKE is possible and this provides choice and opportunity for our users.

Summary

CNI is one of the most competitive sub-spaces in the container technology landscape. Different players with one or more itches to scratch have produced a truly fascinating set of networking projects. What makes this even more extraordinary is that they are all adhering to the CNI specifications and in some cases, co-exist rather well too and Oracle is now one of them with NPN.

In the coming weeks, we’ll be looking at NPN in more detail to help users with planning and adoption while finishing its inclusion in Terraform OKE module . For now, I hope you enjoy this piece on Cilium.