How to uninstall Multus CNI without borking your K8s cluster

What is Multus CNI?

Multus CNI is an open-source Container Network Interface. Unlike other popular Kubernetes CNI projects, such as Cilium, Calico and Flannel, Multus is designed natively to provide advanced Kubernetes networking capabilities, particularly multi-network support.

That means it enables your Kubernetes pods to connect to multiple networks by integrating multiple CNIs into a single Kubernetes cluster.

Multus’s multi-network support is particularly relevant for scenarios such as NFV (Network Function Virtualization) use cases or complex multi-tenant environments.

At Spectro Cloud, we use Multus in our Virtual Machine Orchestration (VMO) capability, and you can add it to your clusters (or any of the other CNIs mentioned above) through your Palette Cluster Profiles.

Why do you need to remove Multus from a cluster?

Most guides and readmes focus on how to install a piece of software — so it’s easy enough to deploy Multus! But there are several reasons why you might need to uninstall CNI like Multus from your clusters.

You might be:

• Switching CNIs: Adopting a different CNI that better fits your evolving networking requirements.
• Decommissioning a cluster: Preparing a cluster for a new use case that doesn’t require the current CNI.
• Troubleshooting: Addressing persistent network issues by removing and reinstalling the CNI to achieve a clean state.

In our use case, we needed a reliable way to uninstall Multus, without destroying our current cluster.

Uninstalling Kubernetes CNIs is not easy!

Kubernetes doesn’t provide a standard method for uninstalling CNIs, leaving users to manage the uninstallation process themselves if they ever choose to switch CNIs.

CNIs integrate deeply with container runtimes, so removing them can be complex. Unlike installing a CNI, which is well-documented and supported, uninstalling often lacks clear guidelines.

This lack of a standardized uninstallation process can be particularly troublesome for complex CNIs like Multus, which integrates multiple network interfaces into pods.

Entering a standard `kubectl delete multus-daemonset.yaml` from the command line will leave residual config files dangling that’ll leave your cluster in a broken state, making pod creation impossible.

Here’s what typically happens:

Errors on new pods

After deleting the Multus with `kubectl delete -f multus-daemonset.yml`, and then attempting to apply any new pod, you’ll encounter failures like this on the new pod:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e5c6dfb35b19f6fc8c7edeaa1855d62800bb64959e03104a1e4c17343d3390bf":  plugin type=”multus” name=”multus-cni-network” failed (add): Multus: [default/my-pod/b023812f-fc48-4ec7-9cdf-4f480f643c59]: error waiting for pod: Unauthorized

This error indicates that Kubernetes is still trying to use Multus configurations that should have been removed. Once this happens, the new pod will never end up actually starting up successfully and will be stuck with the ContainerCreating status.

Dangling files are to blame

The primary culprits for this issue are the lingering configuration files and binaries that Multus installs, particularly:

• /etc/cni/net.d/00-multus.conf
• /opt/cni/bin/multus

These files need to be manually removed to fully uninstall Multus and restore normal pod operations.

Manual cleanup works, but it’s slow

The solution involves a thorough cleanup of these residual files. This can be done manually, but it’s error-prone and time-consuming.

What if there was a way to automate this cleanup process?

It’s time to introduce spectro-cleanup.

Introducing spectro-cleanup

Spectro-cleanup is an open source tool developed by Spectro Cloud.

It provides an easy way to cleanup arbitrary files from nodes and resources from a K8s cluster.

It’s a general-purpose tool, but we can use it to target the cleanup of the dangling files left after Multus is uninstalled, ensuring our cluster is in a good state.

Spectro-cleanup can be run as either a DaemonSet or a Job, which provides the flexibility to perform cleanup operations without needing to install additional software on the nodes.

Here’s a brief overview of its components:

• ConfigMap: Defines which files and resources to clean up.
• DaemonSet/Job: Executes the cleanup based on the ConfigMap’s instructions.
• ServiceAccount and RBAC: Grants necessary permissions to allow spectro-cleanup to perform the cleanup.

Once the files we’ve chosen to delete are cleaned up, spectro-cleanup can then delete the configmaps used to define the files/resources to be deleted and finally delete the daemonset or job used to execute the cleanup.

Using spectro-cleanup to properly uninstall Multus

To uninstall Multus CNI properly using spectro-cleanup, follow these steps:

Step 1: Configure spectro-cleanup

Create a Kubernetes Job or DaemonSet to run spectro-cleanup. Below is the YAML configuration tailored for Multus cleanup via a Job:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spectro-cleanup-role
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - serviceaccounts
  verbs:
  - '*'
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - '*'
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  - roles
  verbs:
  - '*'
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: spectro-cleanup-rolebinding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: spectro-cleanup-role
subjects:
  - kind: ServiceAccount
    name: spectro-cleanup
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: spectro-cleanup
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: spectro-cleanup-config
  namespace: kube-system
data:
  # multus files we want to delete
  file-config.json: |-
    [
      "/host/etc/cni/net.d/00-multus.conf",
      "/host/opt/cni/bin/multus"
    ]
  # spectro-cleanup resources we want to delete
  resource-config.json: |-
    [
      {
        "group": "",
        "version": "v1",
        "resource": "configmaps",
        "name": "spectro-cleanup-config",
        "namespace": "kube-system"
      },
      {
        "group": "batch",
        "version": "v1",
        "resource": "jobs",
        "name": "spectro-cleanup",
        "namespace": "kube-system"
      }
    ]
---
apiVersion: batch/v1
kind: Job
metadata:
  name: spectro-cleanup
  namespace: kube-system
spec:
  template:
    metadata:
      labels:
        name: spectro-cleanup
    spec:
      restartPolicy: Never
      serviceAccountName: spectro-cleanup
      containers:
      - name: spectro-cleanup
        image: gcr.io/spectro-images-public/release/spectro-cleanup:1.2.0
        command: ["/cleanup"]
        env:
        - name: CLEANUP_DELAY_SECONDS
          value: "10"
        resources:
          requests:
            cpu: "10m"
            memory: "25Mi"
          limits:
            cpu: "20m"
            memory: "50Mi"
        securityContext:
          privileged: true
        volumeMounts:
        - name: spectro-cleanup-config
          mountPath: /tmp/spectro-cleanup
        - name: cni-bin
          mountPath: /host/opt/cni/bin
        - name: cni
          mountPath: /host/etc/cni/net.d
      volumes:
        - name: spectro-cleanup-config
          configMap:
            name: spectro-cleanup-config
            items:
            - key: file-config.json
              path: file-config.json
            - key: resource-config.json
              path: resource-config.json
        - name: cni-bin
          hostPath:
            path: /opt/cni/bin
        - name: cni
          hostPath:
            path: /etc/cni/net.d

Step 2: Run spectro-cleanup

Deploy the job using kubectl:

kubectl apply -f spectro-cleanup-job.yml‍

Note: This step NEEDS to be run prior to uninstalling Multus. If you already ran the command to delete Multus before this step, then spectro-cleanup will fail to start, due to the same errors that we discussed earlier in this blog post.

Step 3: Uninstall Multus

Proceed to uninstall Multus by deleting the DaemonSet:

kubectl delete -f multus-daemonset.yml

Ensure that all Multus components are removed.

Step 4: Verify by Creating a New Pod

Create and deploy any test pod to verify that it can actually start:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest

kubectl apply -f my-pod.yml

Finally, check the pod status and logs to confirm it starts without errors.

From cleanup to Validator

Spectro-cleanup is a standalone tool, but it’s also integrated into our validator ecosystem. The Validator ecosystem, including plugins like OCI, Network, and AWS, helps validate rules against your environment and potentially alert on any validation failures.

• Validator Plugins: Validates user configured rules against specific cloud or infrastructure components (e.g., OCI, AWS), and then generate ValidationResult custom resources
• Validator: Manages installation of validator-plugins and consumes ValidationResults, publishing them to the sink of your choice (e.g. Slack, Alertmanager)

Spectro-cleanup is used by the validator to automate the removal of all plugin and validator resources when validator is uninstalled.

See the reference code here.

Wrapping up

Uninstalling Multus CNI without proper cleanup can cause significant disruptions to your Kubernetes cluster.

By using spectro-cleanup, you can automate the removal of residual files, ensuring a smooth and error-free uninstallation process.

Follow the steps outlined in this guide to maintain a clean and functional Kubernetes environment.

For more details on spectro-cleanup, visit the spectro-cleanup github repo. Whether you’re managing a small cluster or a large-scale deployment, spectro-cleanup offers a reliable solution for cleaning up any unwanted files or K8s resources.