Force delete Kubernetes namespace

Introduction

This week I was working on removing a Rancher installation. Some time ago we installed it, to see what it could do for us. But, as it turns out, we did not use it.

Although Rancher seems to be an impressive product, there are a few behaviors that did not please us. You can install Rancher using a Helm chart in the default namespace cattle-system. Then it does its own post-install config, resulting in tens of other namespaces. Some of them support different modules of functionality. Others for adding downstream clusters and service accounts to be able to introspect in those clusters.

These are for administrating clusters and rancher projects.

You can do a Helm uninstall to remove the main parts of Rancher. But, that does not remove those other parts. You will need to do that by hand (unless there are some uninstall options that I have missed).

Validating and Mutating Webhooks

Kubernetes supports operator components to react on the creation or mutation of resources by calling a REST service. Kubernetes does this through the concept of validation or mutation webhook configurations.

To see which ones are registered, use the command:

$ kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

For Rancher, this will report:

• validating-webhook-configuration
• mutating-webhook-configuration
• rancher.cattle.io

Apart from the latter one, these names are not very helpful in identifying the components. I just deleted them.

Finalizers

Resources can have so-called finalizers registered under the metadata node:

metadata:
  ...
  finalizers:
  - controller.cattle.io/mgmt-project-rbac-remove
  - clusterscoped.controller.cattle.io/project-namespace-auth_local
...

Those finalizers define a set of conditions to be met before the object can be deleted. They refer to alert controllers that clean up resources owned by the resource that you want to delete. This prevents the deletion of resources with finalizers defined. You can patch a resource to remove these finalizers with the following command:

$ kubectl -n $NS patch $resource_name -p '{"metadata":{"finalizers":null}}' --type=merge

Here, $NS refers to the namespace containing the resource, and $resource_name to the resource including the api-resource group, like: projects.management.cattle.io. It is important to have the full qualified resource name since I found that externalsecrets also refers to clustersecretstores, which are cluster-scoped, and obviously, I did not want those to be deleted.

The option — type=merge, is important because for some custom resources the patch command results in an UnsupportedMediaType-exception, as described in this Kubernetes issue.

Which resources to remove?

You may have encountered this command to get all the resources in a namespace:

$ kubectl -n $NS get all 

The option all in this command does not select the custom resources.

To get all the resources in the namespaces, you can use a command like this:

$ kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -l <label>=<value> -n <namespace>

As taken from Stack overflow — Listing all resources in a namespace.

Using the -l <label>=<value> you can filter on values of particular labels.

The script for forced deletion of a Namespace

First, you’ll need to identify which namespaces you want to remove

Besides the cattle-system namespace, the namespaces created by Rancher are in the range of the following:

• *cattle-*
• …
• c-m-g8xkd8b7
• c-m-qqg6fmkk
• c-m-qswmksl8
• …
• fleet-default
• fleet-local
• …
• local
• …
• p-7mrvl
• p-9lbm6
• p-lhn2n
• …

There are several namespaces with -cattle- in the name and some with fleet- in the name. Then there is the local namespace and the namespaces with the prefixes c- (that Rancher creates for clusters) and p- (for projects). Also, there can be one or more namespaces with user- in the name.

To list the namespaces, I created the script list_namespaces.sh with the command:

kubectl get ns --no-headers -o custom-columns=":metadata.name"

This provides a list with only the names of the namespaces.

Then to delete a particular namespace with all the resources in it, use the script force_delete_ns.sh, providing the namespace as:

$ ./force_delete_ns.sh c-m-g8xkd8b7

The namespace parameter is stored in the global $NS variable. All the functions described below use this variable to only select resources in this namespace.

Count resources

The script has a function count_apiresources() that determines the number of occurences of a api-resource.

Patch resources

The function patch_apiresources() in the script will select all the occurences of a specific api-resource. It then loops over this list to patch each resource to remove the finalizers, as described above.

Delete resources

The function delete_apiresources() is similar to the patch_apiresources() function but deletes the resource with the — force option.

Patch and delete resources

Now, to actually delete a resource, I combined the functions mentioned above into the function patch_delete_apiresources(). This function first determines the number of occurrences of the resources, using count_apiresources(). If the number of rows is greater than 0, it will first patch all the occurrences by calling patch_apiresources(). Since patching a resource may finish an earlier request to delete the resource, the resources are counted again. If there are still resources left, these are deleted using delete_apiresources().

Delete the occurrences of all api-resources

In a for loop on the selection of all Kubernetes api-resources that are namespaced (so no cluster-scoped resources) the function patch_delete_apiresources() is called.

Patch and delete the namespace

Finally, the namespace itself can be patched and removed. Also, a namespace can have finalizers defined. To be sure these are deleted on metadata level, but also the possible finalizers defined in the spec node will be deleted. Maybe not necessary, but also the annotations are deleted, to prevent the namespace is selected in particular operator functionality.

Running the script will give output like:

Run the script

Conclusion

Removing a namespace with a load of (custom) resources can be a tedious job. Especially, when you need to do it for several namespaces. To me, this script helps a lot. Also, it helped me get more experienced in Bash-scripting, 😉.

In my search on cleaning up Rancher, I also found the project: Rancher Resource Cleanup. It will create a job, and accompanying service accounts to run a script that removes all Rancher-related resources, quite like in the script above. But, be careful with it: it also removes istio-system and possibly other resources that you might want to keep.