How to terminate a side-car container in Kubernetes Job

There is a side car pattern in Kubernetes. A pod contains a main container and sub container(s). DB Client and Cloud SQL Proxy. Ruby on Rails container working Puma and Nginx proxy container. It’s useful in most cases but sometimes we’re in trouble by using in k8s Job. Job is one time job. Every containers have to be terminated at the end of execution but proxy containers are going to be alive like daemon processes. In Kubernetes’s thread on Github, many people want to declare side car’s going trigger in manifest file but nowadays we can’t do that. This article is about how I avoid full of zombie jobs.

I wanted a job to execute sql to execute ALTER. I used a MySQL docker container for execute sql through a Cloud SQL proxy docker container to Google Cloud SQL’s MySQL instance. I wrote shell script in the job manifest file like this comment.

apiVersion: batch/v1
kind: Job
metadata:
name: db-migration
spec:
template:
spec:
containers:
- name: client
image: mysql:5.7
command: ["/bin/sh", "-c"]
args:
- |
sleep 2s
trap "touch /tmp/pod/terminated" EXIT
cat /migration/create_db.sql | mysql --defaults-file=/conf/my.cnf --host=$(MYSQL_HOST) $(DB_NAME)
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: cloudsql
key: username
- name: MYSQL_HOST
value: 127.0.0.1
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: cloudsql
key: password
- name: DB_NAME
value: database
volumeMounts:
- mountPath: /migration
name: migration-files
readOnly: true
- mountPath: /conf
name: db-cnf
readOnly: true
- mountPath: /tmp/pod
name: tmp-pod
- image: b.gcr.io/cloudsql-docker/gce-proxy:1.11
name: cloudsql-proxy
command: ["/bin/sh", "-c"]
args:
- |
/cloud_sql_proxy -instances=my-project:us-central1:my-instance=tcp:3306 -credential_file=/secrets/cloudsql/credentials.json & CHILD_PID=$!
(while true; do if [[ -f "/tmp/pod/terminated" ]]; then kill $CHILD_PID; echo "Killed $CHILD_PID because the main container terminated."; fi; sleep 1; done) &
wait $CHILD_PID
if [[ -f "/tmp/pod/terminated" ]]; then exit 0; echo "Job completed. Exiting..."; fi
volumeMounts:
- name: cloudsql-oauth-credentials
mountPath: /secrets/cloudsql
readOnly: true
- mountPath: /tmp/pod
name: tmp-pod
readOnly: true
restartPolicy: Never
volumes:
- name: migration-files
configMap:
name: migration
- name: cloudsql-oauth-credentials
secret:
secretName: cloudsql-oauth-credentials
- name: db-cnf
secret:
secretName: db-cnf
- name: tmp-pod
emptyDir: {}
backoffLimit: 1
parallelism: 1
completions: 1

To say at least, it’s complex. Let’s see section by section.

spec:
template:
spec:
containers:
- name: client
image: mysql:5.7
command: ["/bin/sh", "-c"]
args:
- |
sleep 2s
trap "touch /tmp/pod/terminated" EXIT
cat /migration/create_db.sql | mysql --defaults-file=/conf/my.cnf --host=$(MYSQL_HOST) $(DB_NAME)

Sleep at first for waiting sql-proxy starts. In k8s you can’t set starting order, so there is no choice. You may use loop,

for i in $(seq 20)
do
mysql --defaults-file=/conf/my.cnf --host=$(MYSQL_HOST) $(DB_NAME) -e 'select 1' || (sleep 1; false) && break
done

After that, executes sql and makes a trigger file when EXIT by trap.

Next, let’s see sql proxy’s section.

spec.template.spec.containers:
- image: b.gcr.io/cloudsql-docker/gce-proxy:1.11
name: cloudsql-proxy
command: ["/bin/sh", "-c"]
args:
- |
/cloud_sql_proxy -instances=my-project:us-central1:my-instance=tcp:3306 -credential_file=/secrets/cloudsql/credentials.json & CHILD_PID=$!
(while true; do if [[ -f "/tmp/pod/terminated" ]]; then kill $CHILD_PID; echo "Killed $CHILD_PID because the main container terminated."; fi; sleep 1; done) &
wait $CHILD_PID
if [[ -f "/tmp/pod/terminated" ]]; then exit 0; echo "Job completed. Exiting..."; fi

Line 7, starts cloud sql proxy and set process number. Line 8 waits for trigger file and kills its process. Exit after all.

Honestly, I think it’s easier that you use your own console not k8s, but it’s important that you leave footprint for co-workers that knows you executed some ALTER.