Implementing Logging using Promtail Loki and Grafana on multi-cluster Azure Kubernetes Service Setup

  1. Install the PLG stack on each cluster and visualize the logs from the Grafana instance running on those clusters.
  2. Install PL stack on each logging cluster, but Grafana on just one say monitoring cluster. In this case, all the logs are aggregated by the Loki instances running on individual logging clusters which can be configured as a data source to the Grafana instance.
  3. Install Promtail on each cluster which will push logs to the monitoring cluster where a single Loki/Grafana instance is running. Loki will aggregate all the logs and label them based on the origin. Grafana instance will be used to visualize the logs based on these labels.
  • Maintenance: Solution 3 will have the least maintenance. Just Monitoring cluster with (PLG) components and other clusters with (P) Promtail component. Solution 1 and 2 have many more components to maintain
  • Cost: It is very obvious that solution 3 is most cost- since the compute resources being consumed are the least.
  • Security: Solution 1 exposes Grafana on each cluster which is a security risk as compared to Solution 2/3 where just one monitoring cluster will host Grafana. But Solution 2 will again expose Loki on each cluster to be configured as a data source for Grafana. Hence Solution 3 clearly wins as in this case, Grafana and Loki are exposed on just the monitoring cluster.
  • Promtail scrapes logs from applications running in Log Producer AKS clusters
  • Promtail is configured to discover K8 apps using promtail secret which is used as the promtail config. The same config will contain Loki address to push logs.
  • Grafana is configured with Loki as data source
  • Grafana and Loki are exposed for external access via Ingress controller like Istio or Nginx
  • Access Grafana using the public IP obtained from the first step
  • Fetch Grafana login credentials from K8 secret named loki-grafana.
  • Explore Loki using Grafana Explore option . Select Loki as data source from the top. This is by default set to http://loki:3100
  • Try to check the Log Browser option for logs by selecting the available labels. try to select a label that is unique across log producer labels.
  • Observe that the data for that label from all the Log Producer clusters can be seen.
  • Now that we have the logs getting aggregated and viewed in the monitoring cluster from each log producer clusters, there is a challenge on how to identify which logs correspond to which cluster on Grafana.
  • We all know that each log stream in Loki has unique labels and Promtail has a concept of external labels which can be used to identify logs from each log producer cluster. Refer to this link and look for external labels
  • Promtail helm chart provides an extraArgs property that can be used to specify the external labels. Refer: https://github.com/grafana/helm-charts/blob/main/charts/promtail/values.yaml
  • Based on the values.yaml file above, we can create a local values.yaml file and specify the external-label like below say for log cluster 1.
  • Then upgrade helm using this values file using command on log cluster 1 :
  • Now observe grafana Explore options to view the log browser and look for cluster label log-cluster-1. Observe that selecting this label will show the logs for only log-cluster-1 cluster.
  • The same can be applied to other log clusters with unique labels.

--

--

--

I am a DevOps Engineer and hold Azure Administrator as well Developer Associate Azure certifications. I like exploring Cloud and Devops Technologies.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Connect Excel with Python

Develop our own operating system! #part06

How StarCrossed Uses the Crowdbotics Dating Blueprint to Help You Find “Love at First Sign”

Docker CLI Made Easier

Memoization of Factorials in Python

Symfony Performance tips and tricks

15 great iOS newsletters you should know about

Create Your Own Public URL for Your Localhost (Similar to ngrok)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Abhishek Kumar

Abhishek Kumar

I am a DevOps Engineer and hold Azure Administrator as well Developer Associate Azure certifications. I like exploring Cloud and Devops Technologies.

More from Medium

Hardening Istio security with SPIRE

Integrating OPA with Emissary Ingress on Styra DAS

With Kubernetes, be careful you dont drown in complexity..

The Tuesday Microsoft Patch announcement on April 12, 2022, surprised everyone with an RPC exploit.