Centralized Logging and Monitoring with Kubernetes
The purpose of this article is to give a general introduction to what is expected when trying to set up the Kubernetes infrastructure.
Introduction
Cloud computing adoption is expected to rise in the next couple of years with most organizations retaining 40% of their IT infrastructure on-premises.
NIST defines cloud computing as a model for enabling ubiquitous, convenient, on-demand network access to computing resources rapidly provisioned and released. These can be attained through either using Virtual Machines (VMs) or containers.
Virtualization is a technology where a virtual version of machine hardware, storage devices, and network devices are created using some emulators called Hypervisors (Maheshwari et al., 2018).
Containers, which, unlike VMs, are deployed on the Virtual Environments (VEs), provide faster resource allocation in comparison to virtual machines. How then, do you orchestrate the deployment of these containers? This is where Kubernetes comes in, which is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation (Cloud Native Computing Foundation, 2019). This article aims to highlight how to centrally monitor logs within a Kubernetes cluster.
Why is Centralized Monitoring and Logging Important?
Metrics and logs are critical for any system or application.
- They provide insight into what the system/application is doing.
- Having a central place to observe a multitude of metrics/log sources from multiple endpoints becomes easy.
- Data retention for analysis, troubleshooting, and forensics.
How does a centralized Monitoring and Logging Platform work?
- Have a mechanism to collect all logs from an application.
- Once the logs have been created, they are sent to a central storage location.
- A storage mechanism needs to be implemented to store the logs for further analysis.
Conceptual Model
- Container Evolution.
Fig 1.0 below shows the advancement of technology from traditional deployment to container deployment. This gives a clear overview of the main infrastructural differences that exist between a virtualized deployment (using VMs) and a containerized deployment (using containers).
2. Centralized Logging and Monitoring with EFK
Kubernetes cluster generates the logs, Fluentd is the component that will oversee collecting and aggregating raw data from the Kubernetes cluster and sending it to a central storage location, Elasticsearch will be responsible for storing the logs while Kibana will be used for visualizing the data stored in Elasticsearch.
Deployment
There are several ways to achieve the deployment of this infrastructure. Several implementation methods can be used in terms of virtualization, i.e., different VMs can be deployed using Type 2 hypervisors such as VirtualBox, VMware Workstation, and Player or through using Type 1 Hypervisors such as VMware’s ESXi, KVM (Kernel-Based Virtual Machine), Microsoft Hyper-V and Citrix Hypervisor.
Another way to deploy the infrastructure is by leveraging cloud computing. The VMs can be deployed on GCP (Google Cloud Compute), Microsoft Azure, and AWS (Amazon Web Services).
Several articles are available online to guide users on how to create such infrastructure through both virtualization and cloud computing.
