10 Kubernetes Best Practices to Know

7 min readJan 29, 2023

1. Use Namespaces

Namespaces are actually virtual cluster inside the physical cluster . Namespaces provides logical isolation between the teams and their environment. If your Kubernetes cluster is large and multiple teams are working on it, you need to have separate namespaces for each team. For example, you should create different namespaces for development, testing and production teams. This way, the developer having access to only the development namespace won’t be able to make any changes in the production namespace, even by mistake.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: test
  labels:
    name: mypod
spec:
  containers:
  - name: mypod
    image: nginx

2. Set Resource Requests & Limits

Resource Requests and limits are the mechanisms Kubernetes uses to control resources such as CPU and memory. Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource. Limits, on the other hand, make sure a container never goes above a certain value. The container is only allowed to go up to the limit, and then it is restricted.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        resources:
            limits:
              memory: 200Mi
              cpu: 1
            requests:
              memory: 100Mi
              cpu: 100m
        ports:
        - containerPort: 80

3. Use readiness and liveness probes

Liveness Probe — monitors the availability of an application while it is running. If a liveness probe fails, Kubernetes will restart your pod. This could be useful to catch deadlocks, infinite loops, or just a “stuck” application.
readiness probe - monitors when your application becomes available. If a readiness probe fails, Kubernetes will not send any traffic to the unready pods and the endpoint controller removes the pod from all services. This is useful if your application has to go through some configuration before it becomes available, or if your application has become overloaded but is recovering from the additional load. By having a readiness probe fail, your application will temporarily not get any more traffic…

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  template:
    metadata:
      labels:
        app: my-test-app
    spec:
      containers:
     —name: my-test-app
        image: nginx:1.14.2
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          successThreshold: 3

4. Use Role-based access control (RBAC)

Using RBAC in your k8s cluster is essential to properly secure your system. Users, Groups, and Service accounts can be assigned permissions to perform permitted actions on a particular namespace (a Role), or to the entire cluster (ClusterRole). Each role can have multiple permissions. To tie the defined roles to the users, groups, or service accounts, RoleBinding or ClusterRoleBinding objects are used.

RBAC roles should be set up to grant using the principle of least privilege, i.e. only permissions that are required are granted. For example, the admin’s group may have access to all resources, and your operator’s group may be able to deploy but not be able to read Secrets.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-only
  namespace: default
rules:
- apiGroups:
  - ""
  resources: ["*"]
  verbs:
  - get
  - list
  - watch

5. Monitor Your Cluster’s Resources usage and Audit Policy Logs

Audit logs ensures your Kubernetes remain secure and away from unauthorized access. It includes the requested URL path, the HTTP method, and the user information sending the request.

Kubernetes audit logs are turned off by default. The default settings allow users to store Kubernetes events in the backend database for up to one hour. But it can be insufficient when you are looking to track unauthorized access.

The kube-apiserver is responsible for auditing a Kubernetes cluster. So whenever a request creates a namespace, it is sent over to it for logging and auditing.

It means that by integrating Kubernetes audit logs, you can figure out the answers to the following –

What happened?
Who did it?
When did it happen?
What was the location of such an occurrence?

Monitoring refers to logging the data related to a system’s resource consumption and performance output. In the case of Kubernetes, monitoring aims to ensure that your clusters are performing to the best of their capacities and the end-users are facing a smooth experience while using your product.

Monitoring the components in the K8s control plane is important to keep resource consumption under control. The control plane is the core of K8s, these components keep the system running and so are vital to correct K8s operations. Kubernetes API, kubelet, etcd, controller-manager, kube-proxy and kube-dns make up the control plane.

Control plane components can output metrics in a format that can be used by Prometheus, the most common K8s monitoring tool.

6. Organize Your Objects with Labels

Labels are metadata that can be attached to Kubernetes objects like pods, nodes, and deployments. Key-value pairs can be used to identify and organize these objects in various ways. For example, a label might be used to specify the environment that a particular pod is running in (e.g., “test”, “stage”, or “prod”). It can also be used to indicate the type of service that a deployment is providing (e.g. “frontend”, “backend”, or “database”).

By using Kubernetes labels correctly, DevOps teams can more quickly troubleshoot issues along the application development pipeline, apply configuration changes en masse, and solve cost monitoring, allocation, and management challenges.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  labels:
    environment: test       # descriptive and meaningful name of environment (like dev,test,prod)
    service-type: frontend  # descriptive and meaningful name of service type(like frontend ,backend,database)
    app_version: v1.0       # consistent naming convention and value of application version
spec:
  containers:
  - name: my-container
    image: my-image

7. Upgrade kubernetes Version

You must always have the latest stable version of Kubernetes in your production cluster. The new releases have many updates, additional features, and most importantly, patches to previous version security issues. It helps you in keeping your cluster away from vulnerabilities.

8. Use Version Control System for Configuration files

Configuration files should be stored in version control before being pushed to the cluster. This allows you to quickly roll back a configuration change if necessary. It also aids cluster re-creation and restoration.

9. Use Autoscaling

Autoscaling enables resources to scale up only when needed and scale down when traffic subsides.

The Kubernetes autoscaling mechanism uses two layers:

Pod-based scaling — supported by the Horizontal Pod Autoscaler (HPA) and the newer Vertical Pod Autoscaler (VPA).
Node-based scaling — supported by the Cluster Autoscaler (CA).

The Horizontal Pod Autoscaler (HPA) is designed to increase the replicas in your deployments.

As your application receives more traffic, you could have the autoscaler adjusting the number of replicas to handle more requests.

The Vertical Pod Autoscaler (VPA) is useful when you can’t create more copies of your Pods, but you still need to handle more traffic.

As an example, you can’t scale a database (easily) only by adding more Pods.

A database might require sharding or configuring read-only replicas.

But you can make a database handle more connections by increasing the memory and CPU available to it.

Lastly, the Cluster Autoscaler (CA).

When your cluster runs low on resources, the Cluster Autoscaler provision a new compute unit and adds it to the cluster.

If there are too many empty nodes, the cluster autoscaler will remove them to reduce costs.

10. Use Network Policies and a Firewall

Network policies should be implementedto restrict traffic between objects(pods, services etc) in the K8s cluster. By default, all containers can talk to each other in the network, something that presents a security risk if hackers gain access to a container, allowing them to traverse objects in the cluster. Network policies can control traffic at the IP and port level, similar to the concept of security groups in cloud platforms to restrict access to resources. Typically, all traffic should be denied by default, then allow rules should be put in place to allow required traffic.

Example :- Consider an application ola having 3 different components as follows:

ola-ui (frontend)
ola-api (backend)
ola-db (db)

Expected communication between these components are like ui communicates to api and api communicates to db.

But when all these components of ola application are running in a kubernetes cluster, technically ui component can communicate to db by default.

We have to configure network policies that allows only the valid communications between the pods

As well as using network policies to restrict internal traffic on your K8s cluster, you should also put a firewall in front of your K8s cluster in order to restrict requests to the API server from the outside world. IP addresses should be whitelisted and open ports restricted.