Application migration from Docker Compose to Kubernetes. How, why, and what problems we’ve encountered. Part 2
The article may be useful to DevOps engineers and development professionals. In the first part you will learn more about our client Picvario and the reasons for migration.
Launch of Kubernetes Cluster
The entire existing application infrastructure was deployed in cloud service, and no changes were planned there. The stage has come when it was necessary to deploy the Kubernetes cluster. As I wrote earlier, in the “Platform Selection” section, each large cloud provider has a Kubernetes managed service implementation, and the cloud we used was no exception.
In a nutshell, the benefits of the managed services implementation on the cloud provider side include significant interaction simplification with the cluster infrastructure. Also, the responsibility for the control plane configuration and performance is with the cloud provider. As an engineer, I can focus directly on deploying and administering my application in a cluster, setting up K8s system components (Ingress controller, cluster auto scaler, etc.), setting up worker nodes, monitoring, and so on.
The use of such managed services often justifies itself both financially and in terms of service quality. Therefore, the clear decision was to use Managed Service for Kubernetes, instead of deploying and maintaining the cluster by ourselves.
Despite features and differences, managed services for Kubernetes from different cloud providers are equally well suited to address the issue at hand. If the choice has to be made from scratch, I would choose an appropriate cloud considering all the services, and not according to one particular service criteria.
I ran two separate clusters, one for testing and one for production. In the first case, this is a zonal master, and in the second — a regional master replicated in three availability zones. Why use two separate clusters instead of one? First, to ensure complete isolation of different environments from each other. Second, infrastructure changes need to be tested somewhere. For example, roll-out Kubernetes version updates, component updates, or API changes revision for resources. Many people prefer not to separate environments into separate clusters — both approaches have the right to life. There is no single recipe.
Application deployment in Kubernetes. Helm templating
After the Kubernetes cluster deployment and system components installation, it was time to start the application migration to the cluster.
At that time, about 25 containers were described in the docker-compose file. It was necessary to decide how to implement the launch of our application components in Kubernetes conditions. There are several options:
- Describe all component configurations in standard YAML manifests. You need to prepare separate manifests for each environment, as their configuration will be different. The problem with this huge number of config files approach created and their volumes makes further work with them inconvenient.
- Use tools such as Kustomize or Helm. Both solutions make it easier to work with K8s objects. Kustomize allows you to streamline your experience by creating base manifests with common configuration and overlay files containing environment-specific settings. Helm instead of a large number of YAML manifests creates a set of universal templates, in which necessary values are substituted.
I chose Helm because I found it more convenient. The only thing is that it will take some time to master writing templates. Here, studying popular public Helm charts sources helped me.
Now I’ll tell you how components were transferred from Docker Compose to Helm, for clarity, using an example of two containers. Under the cut, their config is in the docker-compose
backend:
env_file:
- backend.env
command: ["gunicorn", "--chdir", "picvario", "--bind", "0.0.0.0:8000", "--workers", "8", "picvario.config.wsgi"]
image: ${DOCKER_REPO:?err}/picvario-wsgi:${BUILD_ID:?err}
restart: always
beat_worker:
command: celery beat -A config.celery -l INFO --workdir /pcvr/picvario/
env_file:
- backend.env
image: ${DOCKER_REPO:?err}/picvario-wsgi:${BUILD_ID:?err}
restart: alwaysThe beauty of Docker Compose is its simplicity. We describe just one entity called to service, the creation of which will spawn a container in the config file. The number of possible abstractions in Kubernetes is many times greater: Pod, Deployment, StatefulSet, DaemonSet, ReplicaSet, Service, Ingress, etc. What kind of K8s objects do I need to turn my containers from docker-compose into?
Briefly about the main abstractions:
Pod — a group of one or more containers. Kubernetes manages the pod, not the container directly. Pod, in turn, sets parameters for container launching.
ReplicaSet — ensures the launch of specified pod number in cluster.
Deployment — a controller that manages the state of Pod and ReplicaSet. Provides functionality to declaratively update these objects.
StatefulSet — manages the deployment of pods set while ensuring that the state of each is maintained. Used for stateful applications.
Service — an abstraction that provides network access to pods, usually within a cluster.
Ingress — object for managing external access to services in the cluster (usually HTTP). Ingress takes care of traffic balancing, SSL termination, and traffic routing through virtual hosts.
We are sorted out with the purpose of abstractions. Now a few words about application components themselves. The backend container runs the Django rest API with Gunicorn as the WSGI server. And in beat-worker Celery Beat process runs, performing periodic tasks on a schedule.
The first step was to create a Deployment object for each of these components. Those, in turn, initiate pods launch, inside which containers work. At the same time, some parameters for these two Deployments will be common, and some will differ — this is where templating comes to the rescue. Instead of describing each object configuration, you can create one universal template, in which necessary parameters from files with variables will be substituted. It turns out such structure:
- Template with resource configuration description.
- File with variables values.yaml. Parameters from it are substituted into the template.
- Files with values-dev.yaml and values-prod.yaml variables. Used to pass environment-specific parameters. Variables from them are substituted into templates and overridden variables from values.yaml
This is what the Deployment template looks like:
{{- range $name, $app := .Values.apps }}
{{- if $app.deployment_enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: "{{ $name }}"
namespace: "{{ $.Release.Namespace }}"
labels:
app: {{ $name }}
app.kubernetes.io/name: {{ $name }}
app.kubernetes.io/component: {{ $name }}
{{ include "common_labels" $ | indent 4 }}
spec:
revisionHistoryLimit: 2
replicas: {{ $app.replicaCount }}
selector:
matchLabels:
app: {{ $name }}
template:
metadata:
annotations:
{{- range $fmap := $app.files }}
checksum/{{ $fmap.name }}: {{ include (print print $.Template.BasePath "/configs/" $fmap.name ".yaml" ) $ | sha256sum }}
{{- end }}
labels:
app: {{ $name }}
spec:
{{- if $app.topologySpreadConstraints_enabled }}
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: {{ $name }}
{{- end }}
{{- if $app.nodeSelector }}
nodeSelector:
env: '{{ $app.nodeSelector }}'
{{ else }}
nodeSelector:
env: '{{ $.Values.nodeSelector }}'
{{ end }}
{{- if $.Values.affinity }}
affinity:
{{- toYaml $.Values.affinity | nindent 8 -}}
{{- end }}
terminationGracePeriodSeconds: 300
{{- if $app.files }}
volumes:
{{- range $fmap := $app.files }}
- name: "{{ $.Release.Name }}-{{ $fmap.name }}"
configMap:
name: "{{ $.Release.Name }}-{{ $fmap.name }}"
{{- end }}
{{- end }}
{{- if $.Values.image.use_auth_secret }}
imagePullSecrets:
- name: regcred
{{- end }}
containers:
- name: {{ $name }}
{{- if and $app.image_tag $app.image_name }}
image: "{{ $.Values.image.registry }}/{{ $app.image_name }}:{{ $app.image_tag }}"
{{- else if $app.image_name }}
image: "{{ $.Values.image.registry }}/{{ $app.image_name }}:{{ $.Values.image.tag }}"
{{- else }}
image: "{{ $.Values.image.registry }}/{{ $.Values.image.name }}:{{ $.Values.image.tag }}"
{{- end }}
imagePullPolicy: {{ $.Values.image.pullPolicy }}
{{- if $app.lifecycle }}
lifecycle:
{{- toYaml $app.lifecycle | nindent 12 -}}
{{- end }}
env:
{{- range $commonkey, $commonval := $.Values.env }}
- name: "{{$commonkey}}"
value: "{{$commonval}}"
{{- end }}
{{- if $.Values.secrets_env }}
{{- range $sname, $skey := $.Values.secrets_env.keys }}
- name: "{{ $sname }}"
valueFrom:
secretKeyRef:
key: "{{ $skey }}"
name: "{{ $.Values.secrets_env.name }}"
{{- end }}
{{- end }}
{{- range $key, $val := $app.env }}
- name: "{{$key}}"
value: "{{$val}}"
{{- end }}
{{- if $app.command }}
command:
{{- range $app.command }}
- {{ . }}
{{- end }}
{{- end }}
{{- if $app.args }}
args:
{{- range $app.args }}
- {{ . }}
{{- end }}
{{- end }}
{{- if $app.files }}
volumeMounts:
{{- range $fmap := $app.files }}
- mountPath: {{ $fmap.path }}
name: "{{ $.Release.Name }}-{{ $fmap.name }}"
readOnly: true
{{- if $fmap.file }}
subPath: {{ $fmap.file }}
{{- end }}
{{- end }}
{{- end }}
resources:
{{- if and $app.resources.cpu_limit $app.resources.memory_limit }}
limits:
cpu: "{{ $app.resources.cpu_limit }}"
memory: "{{ $app.resources.memory_limit }}"
{{- end }}
requests:
cpu: "{{ $app.resources.cpu }}"
memory: "{{ $app.resources.memory }}"
{{- if $app.livenessProbe }}
livenessProbe:
{{- toYaml $app.livenessProbe | nindent 12 -}}
{{- end -}}
{{- if $app.readinessProbe }}
readinessProbe:
{{- toYaml $app.readinessProbe | nindent 12 -}}
{{- end }}
---
{{- end }}
{{- end }}Here are values.yaml. In the file beginning, I specify an image from which containers will be launched. Then I list the deployments themselves and some basic parameters for their:
image:
registry: cr.yandex/something
name: picvario-wsgi
tag: latest
pullPolicy: Always
apps:
backend:
deployment_enabled: true
command:
- /bin/bash
- -c
- gunicorn --chdir picvario --bind 0.0.0.0:8000 --workers 8 --timeout 1200 --graceful-timeout 300 picvario.config.wsgi
beat-worker:
deployment_enabled: true
command:
- /bin/bash
- -c
- celery beat -A config.celery -l INFO --workdir /pcvr/picvario/Now content values-production.yaml
nodeSelector: picvario-production
apps:
backend:
replicaCount: 2
topologySpreadConstraints_enabled: true
resources:
memory: 4G
memory_limit: 4G
cpu: 500m
cpu_limit: 500m
beat-worker:
replicaCount: 1
resources:
memory: 1G
memory_limit: 2G
cpu: 200m
cpu_limit: 200m
env:
PRODUCTION_DEBUG_ENABLED: False
HAYSTACK_URL: http://something:9200/
DEFAULT_LANG: ru
DJANGO_SETTINGS_MODULE: config.settings.prodIt turned out to be a capacious deployment description. I can add a dozen other components here, and the template will remain the same. The creation resulted in two Deployment objects — backend and beat-worker. They have common basic settings but differ in command, resource parameters, and the number of replicas to be launched. The topologySpreadConstraints_enabled: true option is also enabled for the backend, which distributes replicas across different availability zones. Thanks to the use of the mechanism of the if-else condition in the template, this functionality was enabled only for the backend, and beat-worker deployment was created without these parameters.
So the containers are running. Now we need to make the backend pods network accessible within the cluster so that other components can access them. As I wrote above, Service abstraction is responsible for this. At the same time, this wasn’t necessary for a beat-worker, as it doesn`t need incoming connections.
Created a Service template
{{- range $name, $app := .Values.apps }}
{{ if and $app.service.enabled -}}
apiVersion: v1
kind: Service
metadata:
name: "{{ $name }}"
namespace: "{{ $.Release.Namespace }}"
labels:
app: {{ $name }}
app.kubernetes.io/name: {{ $name }}
app.kubernetes.io/component: {{ $name }}
{{ include "common_labels" $ | indent 4 }}
spec:
ports:
{{- range $key, $val := $app.service.ports }}
-
{{- range $pkey, $pval := $val }}
{{ $pkey}}: {{ $pval }}
{{- end }}
{{- end }}
selector:
app: "{{ $name }}"
---
{{- end }}
{{- end }}Added service.enabled option, and also specified port on which application is available
apps:
backend:
deployment_enabled: true
command:
- /bin/bash
- -c
- gunicorn --chdir picvario --bind 0.0.0.0:8000 --workers 8 --timeout 1200 --graceful-timeout 300 picvario.config.wsgi
service:
enabled: true
ports:
- name: http
targetPort: 8000
port: 8000So the backend container from Docker Compose in the realities of Kubernetes turned into Deployment and Service objects, and the beat worker container only into Deployment. By analogy, I deployed the rest of the application containers. It turned out three separate Helm charts for different application components — backend, frontend, and nginx-proxy (more about it later) with the same templates.
External access provision. Ingress
Next, it was necessary to configure external access to the application via HTTPS, to its front-end and back-end parts. Historically, this has worked for us through a scheme with two Nginx. The first one was responsible for SSL termination and processing of outside requests and proxied them to the second Nginx, launched as a docker container. That, in turn, routed traffic between application components, distributed static files, proxied calls to S3, and performed header manipulations and redirects. In the new infrastructure, we didn`t move away from this scheme. As part of a separate Helm chart, I also created Deployment and Service for this container. However, it was necessary to resolve the issue so that external traffic from users reached it. This requires an Ingress.
There are two types of Ingress entities in Kubernetes. Ingress resource contains a set of rules for routing traffic to services within the cluster. Ingress controller runs proxy pods that enforce rules set in Ingress resources.
First of all, Ingress Controller was installed in the cluster. There are a large number of solutions to choose from, I took the most common — Nginx Ingress.
Example of commands for Nginx Ingress installing:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginxDeploying a controller automatically creates a Load Balancer resource in the cloud, which will be an entry point for cluster-accessing applications.
Next, I prepared the Helm template for Ingress resources, described necessary rules in values, and created objects.
ingress.yaml:
{{- if .Values.ingresses }}
{{- range $name, $ingress := .Values.ingresses }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "{{ $name }}"
namespace: "{{ $.Release.Namespace }}"
annotations:
kubernetes.io/ingress.class: nginx
{{- if $ingress.cluster_issuer }}
cert-manager.io/cluster-issuer: '{{ $ingress.cluster_issuer }}'
{{- end }}
{{- if $ingress.annotations }}
{{- toYaml $ingress.annotations | nindent 4 -}}
{{- end }}
{{ include "common_labels" $ | indent 4 }}
spec:
tls:
- hosts:
- '{{ $ingress.domain}}'
secretName: "{{ $name }}"
rules:
- host: '{{ $ingress.domain}}'
http:
paths:
{{- if $ingress.rules }}
{{- toYaml $ingress.rules | nindent 10 -}}
{{- end }}
---
{{- end }}values-production.yaml:
ingresses:
frontend:
domain: "*.picvar.io"
cluster_issuer: "letsencrypt-prod"
annotations:
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
rules:
- backend:
service:
name: nginx-proxy
port:
number: 82
path: /
pathType: Prefix
backend:
domain: "*.api.picvar.io"
cluster_issuer: "letsencrypt-prod"
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 4096m
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "75"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1200"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1200"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-WEBAUTH-USER $subdomain;
send_timeout 20m;
rules:
- backend:
service:
name: nginx-proxy
port:
number: 81
path: /
pathType: PrefixWe need to work over HTTPS, so the next step is to install cert-manager, a tool for Let’s Encrypt certificates issued on Kubernetes.
Example of a command for cert-manager installing:
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yamlNext, I added a chart for nginx-proxy to Helm, a template for cert-manager ClusterIssuer, and variables for it.
certmanager-issuer.yaml:
{{- if $.Values.certmanager.enabled }}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: {{ $.Values.certmanager.email }}
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
cloudflare:
email: {{ $.Values.certmanager.email }}
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
selector:
dnsNames:
{{- if $.Values.certmanager.dnsNames }}
{{- toYaml $.Values.certmanager.dnsNames | nindent 8 -}}
{{- end }}values-production.yaml:
certmanager:
enabled: true
email: some@email
dnsNames:
- '*.picvar.io'
- '*.api.picvar.io'As a result, external traffic was balanced and routed to cluster application components. Thanks to cert-manager configuration, automatic management of SSL certificates and enabling HTTPS appeared, respectively.
Setting up CI/CD process
Kubernetes migration required changes to how updates are delivered to environments. First, I will show you how we organized CI/CD before migration.
As it is clear from the diagram, we used Teamcity as a tool for CI/CD. In general, both Teamcity itself and our historical approach to CI/CD suited everyone. At a minimum, I needed to adapt to Kubernetes deployment, but at the same time, I learned common CI/CD practices in Kubernetes, with the ability to rethink the entire process if necessary. The table below summarizes the advantages and disadvantages that I came to as a result of the research.
In addition to using purely CIOps or GitOps approaches, a combination of these practices is also common. In such cases:
- The image is assembled and pushed to the repository in the CI system.
- A deploy pipeline is running in the CI system, which, as a variable, informs the GitOps operator about the component’s new build. For example, ArgoCD uses the Parameter Overrides mechanism for this, which allows you to control certain variables outside of Git.
- CI system triggers the GitOps operator to synchronize changes.
- The GitOps operator determines that the current application state does not match the one set, and recreates the application with a new image, the tag of which was set in step № 2.
I chose this approach. The advantages obtained when using the GitOps operator compared to classic push deployment through the CI system are very significant. By combining methods, it was possible to level out some of the shortcomings of a particular approach. For example, to leave a scheme of working with Git and CI familiar to us, without the necessity to change processes.
Argo CD, one of the most common tools, was chosen as the GitOps operator. Having installed it in the cluster, I performed all further settings through the tool web interface:
- Set up a cluster connection.
- Created Project and set up access to Git repository with charts.
- Created Applications. Principle — separate application for each Helm chart.
Argo CD supports Helm out of the box. When the Application is created, the path to the desired chart in the repository is specified. Next, the synchronization process starts and Argo CD creates/updates all components described in the chart.
Thus, the issue of automating application deployment to the Kubernetes cluster was resolved. The central tool for building and deploying is still Teamcity, but deployment itself is now done by using ArgoCD. At the same time, despite the upgrade, for our developers and testers, the mechanism for working with git and CI/CD systems hasn`t changed in any way. Convenience for the development team is a significant point in choosing CI/CD approach. It’s important that we managed to comply with it, while not interfering with solving DevOps tasks.
Scalability. Problematic components performance
One of the key goals of Kubernetes moving was automatic application scalability. The more users had an application, the more often the small number of clients created a large application load, and this affected performance of other users. Before moving on to a solution, I’ll tell you why such a problem arose and what exactly it expressed.
Let me remind you that the system is multi-tenant — each client works in an isolated application space, exploiting common system resources. Incoming tasks for background loading and processing of media assets are handled by application workers. At the same time, there are many clients, and queues for processing are common. The queues are built on the FIFO principle. Accordingly, situations arose when one tenant uploaded thousands of photo or video files, and the rest of the clients had to wait until their processing was completed. You can easily imagine users’ dissatisfaction who couldn`t even upload a couple of pictures.
First of all, developers improved an algorithm that manages how the tasks queue is populated. The updated algorithm took into account queue length by tenants and introduced task prioritization. This made it possible to fairly distribute tasks between clients but didn`t improve the performance of the application as a whole.
It was necessary to give the system the ability to adapt to floating load so that all functionality of the application continued to work quickly, and servers did not fail even under heavy load.
The autoscaling mechanism in Kubernetes is great for this. Here is a summary of how it works:
- Scaling pod. The number of its replicas increases under load.
- Scaling node. If current hardware can`t launch new pods, then a new virtual machine is ordered.
Pod scaling:
It is implemented using the standard Kubernetes Horizontal Pod Autoscaler object. For work scaling, I did this:
- Using Helm, I created an HPA object for each pod that needs scaling.
- In HPA parameters, I specified the number of replicas (min/max) and tracked the metric by which scaling occurs.
- Configured Resource Requests parameter for the pod, it is responsible for requesting resources on the node.
Scaling can be set up using metrics. The most basic option is to monitor CPU and/or memory load. HPA also allows you to use custom metrics, for example, from the Prometheus monitoring system. In our case, I chose a metric that tracks the CPU utilization. This is a suitable option for our application, but still not universal for all its components, so I plan to make changes in the future. For Celery workers, it would be more correct to organize scaling based on the number of tasks in the broker’s queue.
Custom metrics allow you to do this.
hpa.yaml:
{{- range $name, $app := .Values.apps }}
{{- if and $app.autoscaler_enabled $app.autoscaler_min $app.autoscaler_max }}
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: {{ $.Release.Name }}-{{ $name }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ $name }}
minReplicas: {{ $app.autoscaler_min }}
maxReplicas: {{ $app.autoscaler_max }}
targetCPUUtilizationPercentage: {{ $app.target_cpuutilization }}
---
{{- end }}
{{- end }}Node scaling:
The Cluster Autoscaler is responsible for automatic adjustment of the number of nodes in service. Depending on the cloud, this component will be installed automatically when configuring node group, or you will need to install it yourself.
Accordingly, if there aren’t enough resources for the pod on existing VMs, then a new virtual machine is provisioned that provides the resources. And as soon as the load subsides, the number of pods decreases, and after it the number of nodes does as well. To specify exactly how many resources the pod needs to allocate, the resource requests parameter is used. Based on this information, the Kube scheduler decides where to place the pod in the node.
I’ll tell you about performance problems that appeared at some stage even with configured scaling. It was the import worker to blame. It was responsible for images loading and processing, video conversion (using FFmpeg), and photo/video preview generation. The problem arose with a large number of downloaded videos. Firstly, the transcoding process led to the IOPS sagging of virtual machines, which greatly affected the performance of other application components located on the same nodes. Secondly, the transcoding speed process suffered itself. The conversion of one video was rather slow, even though scaling led to many parallel conversions.
Here is how it was decided:
- Video transcoding tasks are moved to separate Celery workers. This way more flexibility in orchestration is introduced, and transcoding does not affect other worker processes.
- An isolated node group has been created. Worker deploys strictly on nodes from this group. Implemented node selector mechanism usage, so the pod is deployed only on nodes where the desired label is present.
- FFmpeg parameters changed, and 6 processor cores were allocated to one process. Now a single video conversion is done much faster.
- Resource Requests are configured in such a way that only one FFmpeg worker instance will be present per node. Scaling leads to ordering new nodes within an isolated group and placing replicas on it.
Thus, auto-scaling ensured application speed and scalability. A critical problem in service speed under high load was solved.
Conclusion. Migration results and benefits
There are currently just over 120 pods running in the production cluster, including system components. When loaded, their number automatically increases.
The Migration process of Kubernetes required a lot of work. In addition to infrastructure work, the application itself had to be changed.
At the end of the day the question arises, did it make any sense to run a migration like this? Definitely, it did. As a result, we managed to address some critical problems of service quality that were the main reasons the decision to migrate was made. The application speed under load was increased, fault tolerance was implemented, and we no longer had any downtime during releases. And also, the option to address some future challenges as the app grows and develops was introduced.
