How to create a Monitoring Stack using Kube-Prometheus-stack (Part 1)

Published in

Israeli Tech Radar

5 min readJun 14, 2023

Introduction

In this article, we will teach about creating a monitoring stack using Grafana, Prometheus, AlertManager, and integration with Promtail and Loki.

How does the story begin?

By way of introduction, I’m a DevOps engineer at Tikal Knowledge.
My story started when one of Tikal’s customers needed a monitoring solution. So I did some research and discovered that there are so many ways to implement a simple log display.

First, I tested a few options. Each of these options is complicated.
Maybe there isn’t an easy thing…

Prerequisites:

Knowledge of Kubernetes
K3d-cluster or any Kubernetes provider
helm

Goals & Objectives:

Obtain logs and create an easy monitor with Grafana and Prometheus.

Where did these challenges begin?

This began when I saw a lot of configurations and values that needed to be defined. Until I found the Kube-Prometheus-stack and tried this way
YES! It works great, more than anticipated, and is easy to install.

A little peek:

What do I need to do?

Create a simple k3d cluster locally:
A single-node Kubernetes cluster with a network disabled called monitoring.

k3d cluster create monitoring --api-port 6550 --agents 1 --k3s-arg "--disable=traefik@server:0" --k3s-arg "--disable=servicelb@server:0" --no-lb --wait

Intro to Kube-Prometheus-stack:

Kube-Prometheus-stack, also as Prometheus Operator, is a popular open-source project providing complete monitoring and alerting solutions for Kubernetes clusters. It combines tools and components to create a monitoring stack for Kubernetes environments.

Will use the official Helm Chart Kube-Prometheus-stack,
with a customized value file.

To deploy the monitoring stack, follow these steps:

Creating Namespace called monitoring

kubectl create ns monitoring

Add a new Helm Repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Edit the Values

These settings indicate that the selectors mentioned
(rule, service monitor, pod monitor, and scrape config) will have independent configurations and will not be based on Helm graphic values.

    ruleSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false
    scrapeConfigSelectorNilUsesHelmValues: false

Please use these values.

Run the Helm Deployment Command.

helm upgrade --install -f values.yaml kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring

After we deploy the Kube-Prometheus stack, we get as default apps:

Grafana
PrometheusAlert
Manager.

Testing

Check if all work, I am not bluffing you.

kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring

You’ll have to look at the Prometheus and Grafana dashboard.

Prometheus Dashboard from Kube-Prometheus-Stack Helm Chart

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:3000 -n monitoring

Grafana Dashboard from Kube-Prometheus-stack Helm Chart

Check the pods are running with a command:

kubectl get pod -n monitoring

Create a self a ServiceMonitor

The ServiceMonitor defines an application that scrapes metrics from Kubernetes.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-self
  labels:
    app: kube-prometheus-stack-prometheus
spec:
  endpoints:
  - interval: 30s
    port: web
  selector:
    matchLabels:
      app: kube-prometheus-stack-prometheus

Run ServiceMonitor manifest in this command:

kubectl apply -f servicemonitor.yaml -n monitoring

Integration of Promtail and Loki to Grafana

Intro to Loki:

Loki uses Promtail as its primary log collector, but it can also accept log flows from other sources, such as Syslog or other log senders.
It offers high uptime and supports data replication across multiple replicas for durability and fault tolerance. Loki is tightly integrated with Grafana, allowing you to visualize log metrics.

Add a new Helm Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Run the Helm Deployment Command.

helm upgrade --install -f loki-distributed.yaml loki grafana/loki-distributed -n monitoring

After we deploy Loki, we need to update the Kube-Prometheus-stack values file to add additional Data Sources.

This configuration defines a default additional data source that connects to a Loki log aggregation system. It specifies the URL for accessing Loki and sets the access method to the proxy.

  additionalDataSources: 
    - name: Loki
      type: loki
      url: http://loki-loki-distributed-gateway.monitoring.svc.cluster.local
      access: proxy
      isDefault: true

Please use these values.

Intro to Promtail:

Promtail can log files from different sources, including local files, container output streams (stdout and stderr), and systemd journals.
It supports the development, filtering, and relabeling of logs before sending them to Loki.

  clients:
    - url: http://loki-loki-distributed-gateway.monitoring.svc.cluster.local/loki/api/v1/push

Please use these values.

Run the Helm upgrade Command.

helm upgrade --install -f promtail.yaml promtail grafana/promtail -n monitoring

After we have done deploying Promtail and Loki,
two new data sources will be presented to Grafana.

Run the command:

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:3000 -n monitoring

Connect to Grafana > Menu > Data Sources

Conclusion:

After we researched the issue, I discovered the Kube-Prometheus-stack
and it’s not that bad at all. We can implement it easily with a simple setup and very few parameters.

Thank you for Reading, see you in the next post. 🤟