Implement complete LGTM Monitoring Stack from scratch

Peoples are using old stack for monitoring like ELK(Elastic Search, Logstash and Kibana) which is very costly and hard to maintain if you are going to increase the size.

Recently Grafana community introduces it own stack i.e. LGTM (Loki, Grafana, Mimir and Tempo).

Loki Grafana and Mimir stack is a combination of three powerful open-source tools designed for log aggregation, visualization, and alerting in the field of observability and monitoring. Each component plays a specific role in the stack:

1. Loki: Loki is a horizontally scalable, highly available, and multi-tenant log aggregation system. It is optimized for cost-effective storage and retrieval of logs, making it suitable for handling high volumes of log data. Loki stores logs in a compressed and indexed format, allowing efficient querying and filtering. It integrates seamlessly with other tools and platforms commonly used in observability ecosystems.

2. Grafana: Grafana is a popular open-source platform for visualizing time series data. It provides a flexible and intuitive interface for creating dashboards, charts, and graphs to monitor and analyze various metrics, including log data. Grafana supports numerous data sources, including Loki, allowing users to explore and visualize log data alongside other metrics in a unified manner. It also offers advanced features like alerting, annotations, and templating.

3. Mimir: Mimir is an open-source distributed database that acts as a long-term storage solution for Loki. It provides the ability to retain logs for extended periods, even as Loki itself prioritizes recent logs for faster querying and retrieval. Mimir offers durability, scalability, and fault tolerance, ensuring that log data remains accessible and preserved over time. It complements Loki’s capabilities by providing a reliable storage backend for long-term log retention.

Together, the Loki Grafana and Mimir stack enables organizations to centralize log data, gain insights, and troubleshoot issues effectively. Logs from various sources can be aggregated, stored, and visualized in Grafana dashboards, empowering teams to identify patterns, anomalies, and trends. Alerts can be set up based on log events and trigger notifications to address potential problems promptly. The stack offers scalability, flexibility, and extensibility, making it suitable for monitoring diverse infrastructures and applications.

You also need some exporter/agent that scrap logs or metrics and gove it to mimir and loki.

Installation of Grafana, Mimir and Loki by helm

First install helm in your terminal and execute below commands

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Installation of Loki

helm install loki grafana/loki - values values-dev.yaml - version 3.8.0

Install Promtail for Loki which works as daemonset

helm install promtail grafana/promtail - values value-promtail-dev.yaml - version 6.3.0

MIMIR SETUP

helm install mimir grafana/mimir-distributed - version 4.0.0 - values values-dev.yaml

Grafana-agent Setup

MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/v0.27.1/production/kubernetes/agent-bare.yaml NAMESPACE=grafana-agent /bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/v0.27.1/production/kubernetes/install-bare.sh)" | kubectl apply -f -

Kube-state-metrics

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm install kube-state-metrics prometheus-community/kube-state-metrics - version 4.7.0

Grafana Installation:

helm install grafana grafana/grafana -n monitoring - version=6.50.4 - values values.yaml

Note: We can add datasource and all the dashboard from values.yaml which will avoid overhead everytime grafana get restarted and all data is lost.

Conclusion:

Now, we have our installation ready. We can follow official doc if we wanted to go in detail.

I hope this article was informative to you. I would like to hear your thoughts on this post. If you wish to share your opinion about this article, let’s connect and start a conversation on LinkedIn — Dikshant Rai.