Applying django migrations to kubernetes cluster

To be able to apply the django migrations, we will need to setup a kubernetes job. If we are using cloud_sql_proxy then this gets difficult, which is addressed below.

Things to consider

• initContainers can be used but its hard to get the completion status of initContainers and also they need to be cleaned up after
• initContainers can get called for multiple times per pod. we want migrations to be applied only once.
• we can write custom scripts in Dockerfile but then again multiple dockerfiles launching at start of scaling is scary.
• Rollback should be easy incase of failure. Should only continue incase of migration success.

So k8s job seems to be the right choice.

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
  backoffLimit: 4

The above job can be scheduled with

kubectl apply -f job.yaml

status can be searched using

kubectl describe jobs/pi

If we extend this to django, its fairly straight forward, but if this is in google cloud platform then take are to use the below gcp kill context so that the cloud_sql_proxy is killed right after migration. That way the job gets a completion status.

apiVersion: batch/v1
kind: Job
metadata:
  name: myserver-migration
spec:
  parallelism: 1
  template:
    spec:    
      volumes:
        - name: som-myserver-sql
          secret:
            secretName: som-myserver-sql
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
      containers:
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.16
        command: ["/cloud_sql_proxy", "-instances=my-proj:region:myserver=tcp:5499", "-credential_file=/secrets/cloudsql/credentials.json"] 
        volumeMounts: 
          - name: som-myserver-sql  
            mountPath: /secrets/cloudsql
            readOnly: true
          - name: ssl-certs
            mountPath: /etc/ssl/certs
        resources:
          limits:
            memory: "128Mi"
            cpu: "100m"
      - name: myserver-django
        image: localhost:32000/myserver
        command: ["/bin/sh", "-c"]
        args:
          - |
            python manage.py migrate --noinput;
            sql_proxy_pid=$(pgrep cloud_sql_proxy) && kill -INT $sql_proxy_pid;
        securityContext:
          capabilities:
            add:
              - SYS_PTRACE
        imagePullPolicy: Always
      restartPolicy: Never
      shareProcessNamespace: true
  backoffLimit: 4

the above file is missing some env vars, removed for brevity.

we can apply and wait for this job to complete

kubectl apply -f job.yaml 
kubectl wait --for=condition=complete --timeout=10s jobs/myserver-migration
kubectl delete -f job.yaml 

delete is used to cleanup the resources as k8s does not delete the completed jobs. we can safely add these commands before creating a deploy.

• backward compatible migrations and other API constructs to support blue/green deploys is out of scope of this document.

If this was useful, please follow.