移除該死的Terminating Namespace

人品是決定當機與否的關鍵

你有沒有曾經高興的執行你的terraform destroy或是kubectl delete -f 後，發現一直砍不掉，進去一按下

shawnho@jumper:~/pilot/sre/logging$ kubectl get ns
kubectl get NAME              STATUS        AGE
logging           Terminating   43m
logging-system    Active        43m
asm-system        Active        13h
cert-manager      Active        9h
default           Active        14h
gke-connect       Active        13h
istio-system      Active        13h
kube-node-lease   Active        14h
kube-public       Active        14h
kube-system       Active        14h

我就碰到了…原本移除logging只要1~2分鐘的事，結果移了快10分鐘都還卡在Terminating….於是乎…

放了大絕招

shawnho@jumper:~/pilot/sre/logging$ kubectl delete ns logging --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
namespace "logging" force deleted

還是繼續撐著，實在受不了。Namespace卡住，我怎麼部署一個新的同名namespace啊？ 於是上網找了一下，看到了這個哥們的文章，於是快速的測試了一下。

先把namespace output出成 logging.json，把下面一段“spec.finalizers”裡面的kubernetes 文字給移除。

kubectl get ns logging -o json > logging.json
vim logging.json
--
# 字串搜尋
    "spec": {
        "finalizers": [
            ̶"̶k̶u̶b̶e̶r̶n̶e̶t̶e̶s̶"̶
       ]
    },

變成這樣

"spec": {
        "finalizers": [
       ]
    },

之後再透過以下的kubectl replace命令，就可以快速移除Terminating中的namespace了。

kubectl replace --raw "/api/v1/namespaces/logging/finalize" -f ./logging.json
{"kind":"Namespace","apiVersion":"v1","metadata":{"name":"logging","selfLink":"/api/v1/namespaces/logging/finalize","uid":"337e04f0-1ead-4af1-aafb-b530ade4a1e8","resourceVersion":"403725","creationTimestamp":"2021-05-17T14:39:24Z","deletionTimestamp":"2021-05-17T15:17:45Z","labels":{"istio-injection":"disabled"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"istio-injection\":\"disabled\"},\"name\":\"logging\"}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2021-05-17T14:39:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:istio-injection":{}}},"f:status":{"f:phase":{}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-05-17T15:17:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"NamespaceContentRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}}]},"spec":{},"status":{"phase":"Terminating","conditions":[{"type":"NamespaceDeletionDiscoveryFailure","status":"False","lastTransitionTime":"2021-05-17T15:17:56Z","reason":"ResourcesDiscovered","message":"All resources successfully discovered"},{"type":"NamespaceDeletionGroupVersionParsingFailure","status":"False","lastTransitionTime":"2021-05-17T15:17:56Z","reason":"ParsedGroupVersions","message":"All legacy kube types successfully parsed"},{"type":"NamespaceDeletionContentFailure","status":"False","lastTransitionTime":"2021-05-17T15:17:56Z","reason":"ContentDeleted","message":"All content successfully deleted, may be waiting on finalization"},{"type":"NamespaceContentRemaining","status":"True","lastTransitionTime":"2021-05-17T15:17:56Z","reason":"SomeResourcesRemain","message":"Some resources are remaining: datastore.logging.cloud.google.com has 1 resource instances"},{"type":"NamespaceFinalizersRemaining","status":"True","lastTransitionTime":"2021-05-17T15:17:56Z","reason":"SomeFinalizersRemain","message":"Some content in the namespace has finalizers remaining: datastore.logging.cloud.google.com in 1 resource instances"}]}}

訊息的最後，其實會跟我們說這個namespace為何移除不了，像這個感覺起來有datastore的問題。(我在後續安裝時，也確實發生datastore mount不回來的狀況）。所以雖然這個機制 真的能幫大家去除掉討厭的Terminating…但事出必有因，後續能不能正常使用同個namespace 還是看個人修行囉~ :p

[Note] 我真的做人失敗，所以還是乖乖移除cluster重來吧…

提供一小段自動化Script, 讓大家可以快速移除卡住的namespace:

#remove-pending-namespace.sh [pending-namespace-name]
#!/bin/bash
if [[ $# -ne 1 ]]; then
  echo "Please input only namespace name"
  exit 1
fi
ns=$1
kubectl get ns ${ns} -o json > tmp.json
cat ./tmp.json | jq 'del(.spec.finalizers[])' > ./modify.json
kubectl replace --raw "/api/v1/namespaces/${ns}/finalize" -f ./modify.json
rm -f tmp.json modify.json