How to Install Grafana Loki using HELM

Loki is built around the idea of only indexing metadata about your logs labels. Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.

A. Introduction

Loki can be deployed in three different ways depending on the size of your logs and hardware capabilities

• Monolithic
• Simple Scalable
• Microservice

Monolithic

• Description: Single process handling all functions. Easy to set up and manage, suitable for small to medium workloads.
• Use Case: Ideal for small teams or environments with low to moderate log volumes.
• Scalability: Limited scalability

Simple Scalable

• Description: Multiple instances with a straightforward configuration. Allows for more scalability compared to monolithic mode.
• Use Case: Suitable for medium to large environments where log volumes are higher and some level of scaling is required.
• Scalability: Better scalability than monolithic mode.
• Redundancy: Multiple instances provide redundancy and increase reliability

Microservice

• Description: Fully decoupled components, each handling specific functions (ingestion, storage, query, etc.). Allows for horizontal scaling of individual components.
• Use Case: Ideal for large-scale deployments with very high log volumes and complex query requirements.

Loki is capable of handling petabytes of logs a day. 😲

Loki architecture

This is write and read paths on high level

Write path

1. The distributor receives an HTTP POST request with streams and log lines.
2. The distributor hashes each stream contained in the request so it can determine the ingester instance to which it needs to be sent based on the information from the consistent hash ring.
3. The distributor sends each stream to the appropriate ingester and its replicas (based on the configured replication factor).
4. The ingester receives the stream with log lines and creates a chunk or appends to an existing chunk for the stream’s data. A chunk is unique per tenant and per label set.
5. The ingester acknowledges the write.
6. The distributor waits for a majority (quorum) of the ingesters to acknowledge their writes.
7. The distributor responds with a success (2xx status code) in case it received at least a quorum of acknowledged writes. or with an error (4xx or 5xx status code) in case write operations failed.

Read path

1. The query frontend receives an HTTP GET request with a LogQL query.
2. The query frontend splits the query into sub-queries and passes them to the query scheduler.
3. The querier pulls sub-queries from the scheduler.
4. The querier passes the query to all ingesters for in-memory data.
5. The ingesters return in-memory data matching the query, if any.
6. The querier lazily loads data from the backing store and runs the query against it if ingesters returned no or insufficient data.
7. The querier iterates over all received data and deduplicates, returning the result of the sub-query to the query frontend.
8. The query frontend waits for all sub-queries of a query to be finished and returned by the queriers.
9. The query frontend merges the indvidual results into a final result and return it to the client.

B. Installation

1. Add Grafana’s chart repository to Helm:

helm repo add grafana https://grafana.github.io/helm-charts

This whole config file for simple scalable installation

loki:
  auth_enabled: false
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  querier:
    # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
    max_concurrent: 4

deploymentMode: SimpleScalable 
backend:
  replicas: 3 
read:
  replicas: 3
write:
  replicas: 3
# Enable minio for storage
minio:
  enabled: true
# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0
ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

Let’s split it up

loki:
  schemaConfig:
    configs:
      - from: 2024-06-01 #The date from which this schema configuration applies.
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h #The time period for creating new index tables, here 24h (daily).

A critical factor to note here
store: tsdb is Time Series Database, you can read more about it here
object_store: s3which is for Amazon Simple Storage Service (S3) but you change this to

• GCS (gcs): Google Cloud Storage.
• Azure Blob Storage (azure): Azure Blob Storage provided by Microsoft Azure.
• Filesystem (filesystem): Local filesystem storage. Suitable for testing or small-scale setups, not recommended for production.
• Cassandra (cassandra): Apache Cassandra database, suitable for highly scalable distributed storage.
• Bigtable (bigtable): Google Cloud Bigtable.
• DynamoDB (dynamodb):Amazon DynamoDB.
• Swift (swift): OpenStack Swift object storage

ingester:
    chunk_encoding: snappy

In Loki, the chunk_encoding field specifies the type of encoding to use for compressing log data chunks. While snappy is a popular choice due to its balance between compression speed and efficiency, there are other alternatives you can use. Here are the available options:

1. snappy: A fast compression algorithm that offers good compression ratios and high-speed encoding/decoding.
2. gzip: A widely used compression algorithm that provides better compression ratios compared to snappy but at the cost of slower compression and decompression speeds.
3. lz4: An extremely fast compression algorithm that offers faster speeds than snappy but usually at the expense of a slightly lower compression ratio.
4. zstd: A modern compression algorithm that provides a good balance between compression ratio and speed. It often achieves better compression ratios than snappy and lz4, with speeds competitive to snappy.

deploymentMode: SimpleScalable #SingleBinary for monolithic Distributed for microservice

• SingleBinary: All Loki components run in a single process; suitable for small to medium deployments.
• SimpleScalable: Multiple instances with a straightforward configuration for better scalability; suitable for medium to large environments.
• Distributed: Fully distributed microservices architecture with each component running separately; ideal for large-scale, high-volume environments.

Other config values change depending on the deployment mode selected

# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0
ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

By explicitly setting these replicas to zero, the configuration avoids unintended overlap or conflict between different deployment modes, ensuring a clean and efficient deployment aligned with the SimpleScalable architecture.

Apply the values file on your helm command

helm install loki grafana/loki -f values.yml

From the location where you have values.yml.

Your k8s service should look something similar to this

Configure an agent to send data to Loki

Promtail

Get the default config value file

helm show values grafana/promtail > promtail_values.yml

Add the following for clientsclients and scrape_configs

clients:
    - url: http://loki-gateway/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
  - job_name: kubernetes
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: container

The above YAML config shows where Promtail should push its logs http://loki-gateway/loki/api/v1/push

loki-gateway is one the service name that Loki created during installation

Under scrap_configs I have added two jobs systems and kubernetes

You can be able to add as many custom job you need

Grafana

Install grafana

helm install grafana grafana/grafana

Default user: admin
Password

kubectl get secret grafana -o jsonpath="{.data.admin-password}" | base64 --decode

With this in place, you can log-in Grafana

And set the data source to be Loki

You will can be able to generate your logs

C. Conclusion

• ELK Stack: Feature-rich in terms of search, analytics, data processing, and advanced visualization capabilities. It is ideal for use cases that require complex querying, detailed log analysis, and advanced features like machine learning and security.
• Grafana Loki: While not as feature-rich in terms of search and analytics, Loki excels in efficient, scalable, and cost-effective log management. Its integration with Grafana and Prometheus provides a powerful and unified monitoring solution, especially suited for cloud-native environments and Kubernetes.

If your primary needs involve advanced search, detailed log analysis, and powerful data processing, the ELK stack is the more feature-rich choice. If you prioritize efficient log management, scalability, and integration with a unified monitoring stack, Grafana Loki is a strong contender.