How to change expired certificates in kubernetes cluster.

Recently we have faced an issue in kubernetes certificate expiration. kubernetes master node communication is happening through SSL tunneling . SSL tunneling typically relies on a set of trusted third-party certificate authorities to establish the authenticity of certificates.When certificate expired master node communication will stop. Once it happens your system will face temporally downtime. This article will help you to resolve/avoid your certificate expiration issues.

root@xxxxxxxx:/home/ubuntu# kubectl get pods -o wide
Unable to connect to the server: x509: certificate has expired or is not yet valid

Generate new certs

Create new directory called certs

mkdir certs

Create openssl.cnf file inside certs directory with following content(Please change parameters according to your environment)

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = internal-XXXXXXXXXXXXXX.ap-south-1.elb.amazonaws.com
DNS.4 = *.*.compute.internal
DNS.5 = *.ec2.internal
IP.1 = 10.3.0.1

Run following commands to generate certificates.

openssl genrsa -out certs/ca-key.pem 2048
openssl req -x509 -new -nodes -key certs/ca-key.pem -days 365 -out certs/ca.pem -subj "/CN=kube-ca"
openssl genrsa -out certs/apiserver-key.pem 2048
openssl req -new -key certs/apiserver-key.pem -out certs/apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in certs/apiserver.csr -CA certs/ca.pem -CAkey certs/ca-key.pem -CAcreateserial -out certs/apiserver.pem -days 365 -extensions v3_req -extfile openssl.cnf
openssl genrsa -out certs/worker-key.pem 2048
openssl req -new -key certs/worker-key.pem -out certs/worker.csr -subj "/CN=kube-worker"
openssl x509 -req -in certs/worker.csr -CA certs/ca.pem -CAkey certs/ca-key.pem -CAcreateserial -out certs/worker.pem -days 365
openssl genrsa -out certs/admin-key.pem 2048
openssl req -new -key certs/admin-key.pem -out certs/admin.csr -subj "/CN=kube-admin"
openssl x509 -req -in certs/admin.csr -CA certs/ca.pem -CAkey certs/ca-key.pem -CAcreateserial -out certs/admin.pem -days 365

Create certs zip using certs folder

zip -r certs.zip certs

Master Nodes Certificate change

Please ssh into masters and run the commands.

Upload the generated certs.zip file to master nodes and extract it to /tmp

BACKUP /etc/kubernetes DIRECTORY.

cp /etc/kubernetes /home/user/backups

And, then remove all *.pem files from /etc/kubernetes/ssl/

rm /etc/kubernetes/ssl/*.pem

Then run the following commands on all master nodes.

sudo mv /tmp/certs/ca.pem /etc/kubernetes/ssl/ca.pem
sudo mv /tmp/certs/apiserver.pem /etc/kubernetes/ssl/apiserver.pem
sudo mv /tmp/certs/apiserver-key.pem /etc/kubernetes/ssl/apiserver-key.pem
rm -R /tmp/certs
sudo chmod 600 /etc/kubernetes/ssl/*-key.pem
sudo chown root:root /etc/kubernetes/ssl/*-key.pem

Then Restart k8 daemon

sudo systemctl daemon-reload
sudo systemctl start kubelet
sudo systemctl enable kubelet
sudo systemctl stop docker
sudo systemctl start docker

Please make sure docker images age is new .

Running docker ps command will help you to verify docker image age.(Please restart the servers if age is not new)

Worker Nodes Certificate change

Please ssh into worker and run the commands.

Upload the genrated certs.zip file to worker nodes and extract it to /tmp

BACKUP /etc/kubernetes DIRECTORY

cp /etc/kubernetes /home/user/backups

And, then remove all *.pem files from /etc/kubernetes/ssl/

rm /etc/kubernetes/ssl/*.pem

Then run the following commands on all worker nodes.

sudo mv /tmp/certs/ca.pem /etc/kubernetes/ssl/ca.pem
sudo mv /tmp/certs/worker.pem /etc/kubernetes/ssl/worker.pem
sudo mv /tmp/certs/worker-key.pem /etc/kubernetes/ssl/worker-key.pem
sudo chmod 600 /etc/kubernetes/ssl/*-key.pem
sudo chown root:root /etc/kubernetes/ssl/*-key.pem
sudo systemctl daemon-reload
sudo systemctl stop kubelet
sudo systemctl start kubelet
sudo systemctl enable kubelet
sudo systemctl stop docker
sudo systemctl start docker

Please make sure docker images age is new .

Running docker ps command will help you to verify docker image age.(Please restart the servers if age is not new)

Change certificates in manager node

Please ssh into managing server which installed kubctl.

Upload the generated certs.zip file to worker nodes and extract it to /tmp and unzip it.

cat ~/.kube/config/

Sample output

apiVersion: v1
clusters:
- cluster:
 certificate-authority: /root/.kube/certs/ca.pem
 server: https://internal-XXXXXXXXXXXXXX.ap-south-1.elb.amazonaws.com
 name: default-cluster
contexts:
- context:
 cluster: default-cluster
 user: default-admin
 name: default-system
current-context: default-system
kind: Config
preferences: {}
users:
- name: default-admin
 user:
 client-certificate: /root/.kube/certs/admin.pem
 client-key: /root/.kube/certs/admin-key.pem

Backup and remove certificates in above certificate path.

rm /root/.kube/certs/admin.pem /root/.kube/certs/admin-key.pem /root/.kube/certs/ca.pem

Copy new certificates into /root/.kube/certs directory

cp /tmp/certs/ca.pem /tmp/certs/admin.pem /tmp/certs/admin-key.pem /root/.kube/certs

Run kubectl command to very kubectl is working fine

Run following commands to refresh the service accounts as they’ll now contain invalid tokens.

kubectl delete serviceaccount default 
kubectl delete serviceaccount — namespace=kube-system default

Please redeploy all services,replication controllers,pods,namespaces etc.. to avoid restarts.