Planning a production ready kubernetes with fundamental Controllers & Operators — Part 1 — Intro
Background on why I’m writing this series
Part of my role at Tikal requires me to interview candidates for DevOps / SRE / Platform Engineers, nowadays in the center of any modern application we have kubernetes and considering kubernetes has many extensions in the form of controllers and operators, one of the questions I like asking is — give me an example of 5 controllers or operators you used / will use when setting up a new kubernetes cluster, this post will attempt to answer that question as much as possible tackling the core challenges we meet when setting up and running an application on kubernetes.
Kubernetes, the ubiquitous container orchestration system, is celebrated for its robustness and flexibility in managing microservices at scale. At its core, the Kubernetes control loop mechanism continuously monitors the state of a cluster to ensure that the actual state matches the desired state specified by the developer. However, standard Kubernetes alone doesn’t handle every aspect needed for modern cloud-native applications, not to mention a production grade cluster.
So, what is missing with standard kubernetes resources? and, why do we need to extend it ?
To address the gaps in Kubernetes’ core functionalities, it’s crucial to understand the areas where additional tools and controllers are essential. This approach maintains Kubernetes as a lightweight and flexible orchestration platform while empowering users to tailor their systems with further capabilities for comprehensive management and operational efficiency. Here’s an extended overview of what’s missing in standard Kubernetes and how third-party components fill these gaps, this will also be the base of what’s to come in this series.
(1) Secrets Management:
By default, Kubernetes handles secrets by storing them in a key-value store in a base64-encoded format. Notably, base64 is not encryption; it is merely encoding. This approach means that secrets are not securely stored by default and can be accessed by anyone with entry to the cluster’s datastore. To bolster security, additional controllers like the External Secrets Operator are crucial. These integrate Kubernetes with external secret management systems — like AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault — that provide encrypted storage and features such as automated secret rotation.
(2) Configuration Updates:
Kubernetes does not automatically propagate updates in ConfigMaps or Secrets to the pods that use them. Tools like Reloader address this by monitoring for changes in ConfigMaps and Secrets and initiating a rolling update to ensure that applications run with the latest configurations, enhancing automation and deployment efficiency.
(3) Ingress Resource and Controllers:
Kubernetes defines an Ingress resource to manage external access to services within a cluster. However, the Ingress resource alone does not manage traffic — it requires an Ingress controller to route traffic to the appropriate services. This design offers the flexibility to choose from various Ingress controllers, such as NGINX or Traefik, according to specific needs, though it also requires additional setup.
(4) Autoscaling:
Kubernetes supports basic horizontal pod autoscaling, which adjusts the number of pod replicas based on observed CPU utilization or other selected metrics. However, sophisticated autoscaling strategies that consider a wider array of metrics, predictive scaling, and custom metrics require advanced tools like Kubernetes Metrics Server, Horizontal Pod Autoscaler (HPA), and custom controllers that integrate with application performance management (APM) tools.
(5) Monitoring, Logging, and Tracing:
Effective monitoring, logging, and tracing are essential for diagnosing and understanding application behavior and performance within Kubernetes. While Kubernetes offers basic logging through pod logs, more comprehensive monitoring and tracing require integration with systems like Prometheus for metrics collection, Grafana for visualization, and Elasticsearch, Fluentd, and Kibana (EFK) or Loki for logging. Tools like Jaeger or Zipkin are necessary for tracing requests that span multiple services in microservices architectures.
(6) Bootstrap & Disaster Recovery:
Kubernetes does not natively handle disaster recovery scenarios or the bootstrap process of complex applications across clusters. Implementing a robust disaster recovery strategy involves additional tools for backup and restore processes, such as Velero, which can manage state backups and restore in case of failures. For bootstrap, tools like Helm for package management and GitOps methodologies facilitated by operators like Argo CD or Flux help manage application deployment and configuration from initialization to ongoing management in a declarative and controlled manner.
In conclusion, while Kubernetes provides a solid foundation for container orchestration, its design necessitates the integration of additional specialized tools to create a fully functional, secure, and resilient environment. This extensible approach allows organizations to tailor their Kubernetes clusters to meet precise operational requirements and adapt to diverse workloads.
In the next post I would like to start discussing Secrets Management and Configuration rotation — the rest of this series is something I refer to as the “additional controllers which do 80% of the job of making your kubernetes cluster production ready” — why 80, because the remaining 20% are cloud specific.
Hope you find this series informative, feel free to drop me a line letting me know what your thoughts are on this subject.
Yours sincerely, Haggai Philip Zagury.
