Analytics Vidhya
Published in

Analytics Vidhya

Write Kubernetes manifests for python flask app

This post is part of the series Prepare and Deploy python app to Kubernetes

👈 Previous post: Containerizing python flask application

👈 Previous post: Getting to know Minikube

At the end of this series, we will have a fully working flask app in Kubernetes. The app itself that we are using, can be found here: https://github.com/brnck/k8s-python-demo-app/tree/docker

Prerequisites

We are going to use minikube to deploy our application to Kubernetes. If you are not familiar with it, head over to this post to learn more, as we are going to cover topics such as how to use Minikube here.

Defining use case

Before we even start writing Kubernetes manifests, we need to clearly define our application use case for ourselves. It’d be easier to identify what resources we will need to deploy to Kubernetes.

So we have an application which:

  • Handles our app users HTTP requests and returns some content;
  • Has a CLI command which also does something. It could be a one-off job or a cronjob. Let’s say we need a one-off job.

That means we need to:

  • Access the app from the outside of the cluster;
  • Be able to independently scale application;
  • Be able to run a one-off job.

These use cases clearly indicate what workloads we are going to use for our application. Now I am not going to dive deeper into the workloads explanation as I am assuming you are already more or less aware what are the differences between them. In addition to the workloads, we must also add additional resources around them so, f.i. clients would be able to reach our app.

To summarize everything, these are the Kubernetes resources we are going to deploy to our Kubernetes cluster:

Preparing directory and files

Create a directory for manifests and add empty (for now) files:

mkdir k8s-manifests
touch k8s-manifests/namespace.yaml
touch k8s-manifests/deployment.yaml
touch k8s-manifests/job.yaml
touch k8s-manifests/service.yaml
touch k8s-manifests/ingress.yaml

Also, add k8s-manifests folder to .dockerignore file as there is no point in adding manifests into an image. If you have been using Kubernetes for some time you might have noticed that some public vendors use only one file for all the manifests. Now while there is nothing wrong with that as it produces the same result as having everything in separate files and deploying one by one. In fact, having all manifests in one file, helps with the ordering of resources deployment, because namespace must be deployed first before anything else. However, for the sake of clarity and readability, we are going to split files by resource.

Creating namespace manifest

Creating namespaces is usually based on context. The most common ones are probably team or project. You could either create one namespace for the whole development team, and it will be used to deploy all kinds of applications. They might even not be related to each other at all. The other way is to create a namespace only for the project. That is what we are going to do with our app as well. Creating a namespace for a project instead of a team prevents issues like “what if the team collaboratively manages applications with other teams?” from even appearing because from the administrator perspective we can easier control accesses and give them only to the applications that the developer or team needs access to.

Our application is called k8s-python-demo-app. We can drop the k8s- prefix and name our namespace python-demo-app.

Our final namespace.yaml file should look like this:

apiVersion: v1
kind: Namespace
metadata:
name: python-demo-app

Creating deployment manifest

To better understand what things are required for our web part of the application, let’s dive deeper and define the use case, as we did with the whole app. It will help to better understand how to write a deployment manifest.

Starting from the very beginning, a deployment called python-demo-app-web must be created in a namespace python-demo-app. It should use python-demo-app as an image and init as an image tag. Moreover, the container must be started as an app (id 1000) user with no privilege escalation. The container should request 128Mi memory and 100 CPU cycles. Also, resources should be limited to 256Mi and 200 CPU cycles. There should be 2 replicas deployed to Kubernetes. Finally, health checks must be performed. Readiness probe should be constantly checking 8000 port and / endpoint. Liveness probe should look for Gunicorn master process and ensure that it is running properly.

The use case is defined. Now start by translating it to deployment resource manifest.

First, start by adding metadata:

apiVersion: apps/v1
kind: Deployment
metadata:
name: python-demo-app-web
namespace: python-demo-app

Moving on to .spec part. Add replicas: 2 to .spec:

apiVersion: apps/v1
kind: Deployment
metadata:
name: python-demo-app-web
namespace: python-demo-app
spec:
replicas: 2

We also need to add .spec.selector. According to the documentation, the .spec.selector field defines how the Deployment finds which Pods to manage. In this case, you select labels that are defined in the Pod template (app: python-demo-app and role: web). However, more sophisticated selection rules are possible, as long as the Pod template itself satisfies the rule.

Let’s define both of these:

apiVersion: apps/v1
kind: Deployment
metadata:
name: python-demo-app-web
namespace: python-demo-app
spec:
replicas: 2
selector:
matchLabels:
app: python-demo-app
role: web
template:
metadata:
labels:
app: python-demo-app
role: web

Moving further to .spec.template.spec. Before defining the container, let's handle security first. We can do that by adding securitycontext key with values runAsGroup: 1000 and runAsUser: 1000 (Because our image user is app (id 1000)). Defining securitycontext for a pod will make Kubernetes apply that context for all the containers that live in that pod. Proceed by adding container name, image, tag, and container port. Remember the previous lesson with docker? We have talked about gunicorn the default listening port which is 8000. It should be defined and name gunicorn. While naming a port is optional, it will help us later.

With the described changes, the deployment file should now look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
name: python-demo-app-web
namespace: python-demo-app
spec:
replicas: 2
selector:
matchLabels:
app: python-demo-app
role: web
template:
metadata:
labels:
app: python-demo-app
role: web
spec:
securityContext:
runAsGroup: 1000
runAsUser: 1000
containers:
- name: python-demo-app-web
image: python-demo-app:init
ports:
- name: gunicorn
containerPort: 8000

Resources that need to be allocated and limited for a container. By default, workloads can have no resources defined. It means that your application can potentially take over the whole cluster. Imagine having a memory leak in an app. This would create problems in a staging cluster and probably catastrophe in a production cluster. That is why it is usually a very good practice to allocate and limit resources for all the containers. However, this is a very deep topic that requires good knowledge of your application and how it works under a big load when booting up and idling. Setting low limits will result in unnecessary application restarts. On the other hand, do not set too high limits as it is a waste of resources and makes your cluster fewer efficient. For the sake of this is only the guide and that we are learning, let’s not concentrate more here and just define some logical resources requests and limits. As I have mentioned above:

Container should request 128Mi memory and 100 CPU cycles. Also, resources should be limited to 256Mi and 200 CPU cycles

Finally, let’s add liveness and readiness health checks. With the readiness we will check whether our web server responds to HTTP requests with the status 200 and with the liveness we will check the PID of gunicorn the process to ensure it is running. Also, let's give it some initial delay so gunicorn workers can be boot up.

The final deployment manifest should look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
name: python-demo-app-web
namespace: python-demo-app
spec:
replicas: 2
selector:
matchLabels:
app: python-demo-app
role: web
template:
metadata:
labels:
app: python-demo-app
role: web
spec:
securityContext:
runAsGroup: 1000
runAsUser: 1000
containers:
- name: python-demo-app-web
image: python-demo-app:init
ports:
- name: gunicorn
containerPort: 8000
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 200m
readinessProbe:
initialDelaySeconds: 10
httpGet:
port: gunicorn
path: /
livenessProbe:
initialDelaySeconds: 10
exec:
command:
- /bin/sh
- -c
- "pidof -x gunicorn"

Creating service manifest

An abstract way to expose an application running on a set of Pods as a network service. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods and can load-balance across them.

As usual, create metadata first:

apiVersion: v1
kind: Service
metadata:
name: python-demo-app-web
namespace: python-demo-app

Moving on to .spec, selector needs to be defined. It works pretty much the same as in deployment. Service needs to know which pods are eligible for the traffic to be routed to. Even the same labels can be added:

spec:
selector:
app: python-demo-app
role: web

Finally, service needs to know not only to which pod route traffic to, but also to which port of the exposed ports. While it can map any port to a targeted pod port and for convenience, the targetPort is set to the same value as the port field let’s use 80 port for the service, name ithttp and route traffic to gunicorn (yes, we can use a name instead of a port number) port in a pod. If you did everything correctly, your final service manifest should look like this:

apiVersion: v1
kind: Service
metadata:
name: python-demo-app-web
namespace: python-demo-app
spec:
selector:
app: python-demo-app
role: web
ports:
- name: http
port: 80
targetPort: gunicorn

If we deploy the application now, any other application that runs inside a cluster could reach this app by simply using service-name.namespace-name syntax. You can read more about inside cluster DNS resolution here. However, our goal is to let our customers consume the app. That is where ingress comes in

Creating ingress manifest

Ingress is basically HTTP and HTTPS router from outside the cluster to services. Routing is based on rules, so you can use one domain and route to completely different pods based on rules. This has only one route / so we are not going to dive deeper into routing to multiple services through one ingress. Instead, we will configure ingress to route traffic to python-demo-app-web service when HTTP request to Kubernetes comes with a host python-app.demo.com and / endpoint

I want to put a big disclaimer here. There is a bug in the Nginx ingress controller. The one that is going to be used in minikube cluster. That is why old apiVersion will be used (you will get a warning when applying ingress). You can take it as a challenge and deploy the ingress controller with the image version that has this bug fixed.

You know the drill already. Metadata :)

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: python-demo-app-web
namespace: python-demo-app

In the .spec section, we will define the ingress class name which I have already mentioned in a disclaimer. Also, rules array must be defined. We can start by adding our rule with the host python-app.demo.com

spec:
rules:
- host: python-app.demo.com

Host it done. Now we need to add an endpoint and attach service to this rule. Everything is definable under host sibling - http. The final result should look like this:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: python-demo-app-web
namespace: python-demo-app
spec:
rules:
- host: python-app.demo.com
http:
paths:
- path: /
pathType: Prefix
backend:
serviceName: python-demo-app-web
servicePort: http

As key names are pretty self-explainable you have probably already known that this manifest translates to:

All http requests coming through ingress controller with the host python-app.demo.com and an endpoint starting with / must be forwarded to the service named python-demo-app and its port http which (if we look to the service manifest again) should forward traffic to one of the app: python-demo-app, role: web labeled pods and its gunicorn port, which is 8000

Everything for web traffic handling is ready. Let’s create the last manifest for running a one-off job

Creating job manifest

Job is basically a pod or pods which are scheduled to run until successful completion or until they are deleted. F.i. jobs could be used to run database schema updates, do some kind of maintenance task. Our job in this application is very simple — to print out Hello, world from CLI!.

As with all other manifests, described above, a Job needs apiVersion, kind, and metadata fields.

apiVersion: batch/v1
kind: Job
metadata:
name: hello-world-job
namespace: python-demo-app

The .spec.template is the only required field of the .spec. What's more, .spec.template is a pod template. It has exactly the same schema as a pod, except it is nested and does not have an apiVersion or kind. In fact, when we will finish creating this file, you will notice that there are many similarities between deployment and job.

As in deployment let's add labels to template:

spec:
template:
metadata:
labels:
app: python-demo-app
role: hello-world-job

Proceed by adding .spec.template.spec.securityContext. We want this container to run as a user app as well. Have you wondered what happens if the job fails? Does Kubernetes restart the job? Or it just marks it as failed and do nothing? Well according to the documentation:

Job is only appropriate for pods with RestartPolicy equal to OnFailure or Never. (Note: If RestartPolicy is not set, the default value is Always

Or in other words, Kubernetes by default, set restartPolicy as Always. If the job would be deployed without setting an appropriate restartPolicy, deployment itself would fail. To solve that, let's also add restartPolicy: OnFailure

spec:
template:
metadata:
labels:
app: python-demo-app
role: hello-world-job
spec:
securityContext:
runAsGroup: 1000
runAsUser: 1000
restartPolicy: OnFailure

For the containers part, there will be some differences. First, we don’t need to expose any container port as this is a CLI command. Next up we won’t do any readiness checks, because the container does not accept any traffic. Also, for the sake of this guide let's not do liveness check as well as this command will run very quickly, so it's useless for this case.

After applying changes containers part should look like this:

      containers:
- name: python-demo-app-hello-world-job
image: python-demo-app:init
command:
- python3
args:
- cli.py
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 200m

You have probably noticed that there two new keys (command and args that was not present in the deployment). Let me explain what we are doing here. If you are following this guide from the beginning you will probably remember that we have used the same image to run our web server and CLI command. The only difference was that we changed the container entrypoint and cmd. That is what we are doing here as well. More explanation about command and args can be found here. An approach like that is not always good. Most of the time separate image specifically for the CLI would be better, but let's not get into the details as that is out of the scope of this guide.

Our final job should look like this:

apiVersion: batch/v1
kind: Job
metadata:
name: hello-world-job
namespace: python-demo-app
spec:
template:
metadata:
labels:
app: python-demo-app
role: hello-world-job
spec:
securityContext:
runAsGroup: 1000
runAsUser: 1000
restartPolicy: OnFailure
containers:
- name: python-demo-app-hello-world-job
image: python-demo-app:init
command:
- python3
args:
- cli.py
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 200m

Deploying to Kubernetes

It is time to deploy the application to Kubernetes.

First things first, create aminikube cluster. I am going to provision my Kubernetes cluster with virtualbox driver and kubernetes-version v1.20.5. Using the same command as mine is strongly advised, otherwise, some parts might not produce the result we expect. Proceed with different configurations only when you know what you are doing.

minikube start --driver=virtualbox --kubernetes-version=1.20.5

Enable ingress addon:

minikube addons enable ingress

Confirm that cluster is up and running:

kubectl get nodes                                        
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 81s v1.20.5

This guide will not cover how to build an image and push it to a private or public registry. Instead, for the sake of simplicity, we are going to build an image directly on minikube virtual machine. Make sure you are in the application directory in your terminal:

eval $(minikube docker-env)
docker build -t python-demo-app:init .
<...>
Successfully built 22008520508b
Successfully tagged python-demo-app:init

Confirm:

minikube ssh docker images | grep python-demo-app
python-demo-app init 22008520508b About a minute ago 125MB

Everything is ready. Proceed by creating a namespace and confirm it:

kubectl apply -f k8s-manifests/namespace.yaml
kubectl get namespaces
NAME STATUS AGE
default Active 11m
ingress-nginx Active 10m
kube-node-lease Active 11m
kube-public Active 11m
kube-system Active 11m
python-demo-app Active 12s # <-- Here is our namespace

Now deploy the rest of the application:

kubectl apply -f k8s-manifests/deployment.yaml \
-f k8s-manifests/service.yaml \
-f k8s-manifests/ingress.yaml \
-f k8s-manifests/job.yaml

Kubectl will return an output:

deployment.apps/python-demo-app-web created
service/python-demo-app-web created
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/python-demo-app-web created
job.batch/hello-world-job created

I have already mentioned why we are using networking.k8s.io/v1beta1 instead of networking.k8s.io/v1

Let’s check if all resources deployed and running:

kubectl get pods,jobs,service,ingress -n python-demo-app
NAME READY STATUS RESTARTS AGE
pod/hello-world-job-9l9w9 0/1 Completed 0 10s
pod/python-demo-app-web-5f756fbcc-628cd 0/1 Running 0 10s
pod/python-demo-app-web-5f756fbcc-cz5kl 0/1 Running 0 10s
NAME COMPLETIONS DURATION AGE
job.batch/hello-world-job 1/1 3s 10s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/python-demo-app-web ClusterIP 10.100.245.172 <none> 80/TCP 10s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/python-demo-app-web nginx python-app.demo.com 80 10s

Web pods are still booting up. Let’s give it few more seconds. While we are waiting, we can check job pod pod/hello-world-job-9l9w9 as the job is marked as completed:

kubectl logs hello-world-job-9l9w9 -n python-demo-app                         
Hello, world from CLI!

Perfect! The one-off job ran and completed successfully. Getting back to web pods:

kubectl get deployment,replicaset,pods -n python-demo-app
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/python-demo-app-web 0/2 2 0 6m9s
NAME DESIRED CURRENT READY AGE
replicaset.apps/python-demo-app-web-5f756fbcc 2 2 0 6m9s
NAME READY STATUS RESTARTS AGE
pod/hello-world-job-9l9w9 0/1 Completed 0 6m9s
pod/python-demo-app-web-5f756fbcc-628cd 0/1 Running 0 6m9s
pod/python-demo-app-web-5f756fbcc-cz5kl 0/1 Running 0 6m9s

Pods are running, but they are not marked as ready. Something is wrong. We need to inspect one of the pods' logs:

kubectl logs python-demo-app-web-5f756fbcc-628cd -n python-demo-app
[2021-04-22 21:22:32 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2021-04-22 21:22:32 +0000] [1] [INFO] Listening at: http://127.0.0.1:8000 (1)
[2021-04-22 21:22:32 +0000] [1] [INFO] Using worker: sync
[2021-04-22 21:22:32 +0000] [7] [INFO] Booting worker with pid: 7

Nothing strange. How about describing that pod:

kubectl describe pod python-demo-app-web-5f756fbcc-628cd -n python-demo-app
<...>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m5s default-scheduler Successfully assigned python-demo-app/python-demo-app-web-5f756fbcc-628cd to minikube
Normal Pulled 9m3s kubelet Container image "python-demo-app:init" already present on machine
Normal Created 9m3s kubelet Created container python-demo-app-web
Normal Started 9m3s kubelet Started container python-demo-app-web
Warning Unhealthy 3m55s (x30 over 8m45s) kubelet Readiness probe failed: Get "http://172.17.0.3:8000/": dial tcp 172.17.0.3:8000: connect: connection refused

By checking both of the outputs, we can see a problem. gunicorn clearly shows that it listens at localhost while Readiness tries to make an HTTP request to container IP. The solution is to bind gunicorn to all interfaces. This can be solved by adding args section to a container with --bind 0.0.0.0 flag. Of course, there are more ways to solve this problem, but I will leave that to you, in case this is too easy.

As we know from Dockerfile, arguments, provided to entrypoint are app:app. That means it must be added to args as well.

Head over to deployment manifest and add args below image key:

<...>
image: python-demo-app:init
args:
- '--bind'
- '0.0.0.0'
- 'app:app'
ports:
<...>

Redeploy

kubectl apply -f k8s-manifests/deployment.yaml
deployment.apps/python-demo-app-web configured

After 30–60 seconds, pods should become running

kubectl get pods -n python-demo-app                      
NAME READY STATUS RESTARTS AGE
hello-world-job-9l9w9 0/1 Completed 0 22m
python-demo-app-web-6c4cc75ddc-tmxjl 1/1 Running 0 67s
python-demo-app-web-6c4cc75ddc-wb79r 1/1 Running 0 87s

Indeed, it is. There we go. We identified and fixed a bug!

Can we reach our application from outside? Let’s check:

curl -H "Host: python-app.demo.com" $(minikube ip)/
Hello, World!

HTTP request was successful, and the answer was returned! There is nothing else to be done. We may now clean up everything from the cluster:

kubectl delete -f k8s-manifests/deployment.yaml \
-f k8s-manifests/service.yaml \
-f k8s-manifests/ingress.yaml \
-f k8s-manifests/job.yaml

Also, remove namespace:

kubectl delete -f k8s-manifests/namespace.yaml

Conclusion

Congratulations on writing your own Kubernetes manifests. If this was too easy, and you want more, here’s a few tasks that can be done additionally:

  • Modify CLI command to read environment variable called MESSAGE and pass the value using configmap
  • Create gunicorn config file using configmap and mount it to web pod. Make sure bind is passed via config file, not through args

Manifests with the application can be found here. Proceed to the next guide, where we will wrap manifests to helm-charts

--

--

--

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Recommended from Medium

Galaxy Shooter 2D — Web GL

Weekend’s Little Humor 10 Programming Jokes of the week By Muhammad Umair

A step-by-step guide to finding a code mentor

Upload multiple images and compress image in Flutter

Azure Functions helps us to alleviate the frustration of unknown waiting time for thousands of…

Data Quality and Reliability

Sending emails with Alpas web framework — Part 1

ERROR AND FAIL.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Augustas Berneckas

Augustas Berneckas

DevOps

More from Medium

Kubernetes Multiple Watches Using Threads in Python

How to run a Python Program within an AWS EC2 Instance

Use Ansible to customize AWS EMR through bootstrap actions

How to deploy micro-services on kbernetes, hands on tutorial !