Kubernetes Observability: Log Aggregation Using ELK Stack

This article was originally published at https://www.magalix.com/blog/kubernetes-observability-log-aggregation-using-elk-stack

In the old days, all components of your infrastructure were well-defined and well-documented. For example, a typical web application could be hosted on a web server and a database server. Each component saved its own logs in a well-known location: /var/log/apache2/access.log, /var/log/apache2/error.log and mysql.log.

Back then, it was very easy to identify which logs belonged to which servers. In a highly complex environment, for example, you could have four web servers and two database engines, which are part of a cluster.

Let’s fast forward to the present day where terms like cloud providers, microservices architecture, containers, ephemeral environments, etc. are part of our everyday life. In an infrastructure that’s hosted on a container orchestration system like Kubernetes, how can you collect logs? The highly complex environment that we mentioned earlier could have dozens of pods for the frontend part, several for the middleware, and a number of StatefulSets. We need a central location where logs are saved, analyzed, and correlated. Since we’ll be having different types of logs from different sources, we need this system to be able to store them in a unified format that makes them easily searchable.

Now that we have discussed how logging should be done in cloud-native environments, let’s have a look at the…