Grafana LOKI-3 target Architecture on EKS 1.27

Loki has introduced a 3 target architecture that makes the workflow more scalable with the read, write, and backend pods. This article guides you through the configuration and setup of the LOKI on the EKS Cluster.

Helm chart used

Grafana Loki: https://artifacthub.io/packages/helm/grafana/loki

LOKI Chart Version: 5.20.0

Grafana Promtail: https://artifacthub.io/packages/helm/grafana/promtail

Promtail chart Version: 6.15.1

Pre-requisites

• Helm
• EKS Version 1.27
• EBS CSI Driver installed(For Persistent Volumes)
• IAM access

LOKI Configuration and Setup

auth_enabled has to be configured as false

# Should authentication be enabled
auth_enabled: false

Create s3 buckets and configure the long-term storage

• observability-loki-chunks-bucket
• observability-loki-ruler-bucket
• observability-loki-admin-bucket

  storage:
    bucketNames:
      chunks: observability-loki-chunks-bucket
      ruler: observability-loki-ruler-bucket
      admin: observability-loki-admin-bucket
    type: s3
    s3:
      # s3: null
      endpoint: null
      region: us-east-1
      secretAccessKey: null
      accessKeyId: null
      signatureVersion: null
      s3ForcePathStyle: false
      insecure: false
      http_config: {}

Configure IAM and Service Account

The Service account has to be created as the Loki pods will access the S3 Buckets with the help of the service account and IAM role.

Configure IAM Role

Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LokiStorage",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::observability-loki-chunks-bucket",
                "arn:aws:s3:::observability-loki-chunks-bucket/*",
                "arn:aws:s3:::observability-loki-ruler-bucket",
                "arn:aws:s3:::observability-loki-ruler-bucket/*",
                "arn:aws:s3:::observability-loki-admin-bucket",
                "arn:aws:s3:::observability-loki-admin-bucket/*"
            ]
        }
    ]
}

Trust-Relation for the Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::acc_id:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/0D803EC8EB3AC675B6B9A794C25E84E8"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/0D803EC8EB3AC675B6B9A794C25E84E8:sub": "system:serviceaccount:loki:eks-loki-access-s3-role-sa",
                    "oidc.eks.us-east-1.amazonaws.com/id/0D803EC8EB3AC675B6B9A794C25E84E8:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Configure the Service account in the helm chart

serviceAccount:
  # -- Specifies whether a ServiceAccount should be created
  create: true
  # -- The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name: eks-loki-access-s3-role-sa
  # -- Image pull secrets for the service account
  imagePullSecrets: []
  # -- Annotations for the service account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::acc_id:role/test-loki-001-demo
  # -- Labels for the service account
  labels: {}
  # -- Set this toggle to false to opt out of automounting API credentials for the service account
  automountServiceAccountToken: true

Set the HELM Test as False

# -- Section for configuring optional Helm test
test:
  enabled: false
  # -- Address of the prometheus server to query for the test
  prometheusAddress: "http://prometheus:9090"
  # -- Number of times to retry the test before failing
  timeout: 1m
  # -- Additional labels for the test pods
  labels: {}
  # -- Additional annotations for test pods
  annotations: {}
  # -- Image to use for loki canary
  image:
    # -- The Docker registry
    registry: docker.io
    # -- Docker image repository
    repository: grafana/loki-helm-test
    # -- Overrides the image tag whose default is the chart's appVersion
    tag: null
    # -- Overrides the image tag with an image digest
    digest: null
    # -- Docker image pull policy
    pullPolicy: IfNotPresent
# Monitoring section determines which monitoring features to enable

Set the monitoring section to False

monitoring:
  # Dashboards for monitoring Loki
  dashboards:
    # -- If enabled, create configmap with dashboards for monitoring Loki
    enabled: false

Set the Recording Rules to false

  rules:
    # -- If enabled, create PrometheusRule resource with Loki recording rules
    enabled: false

Configure the Service Monitor to false

  serviceMonitor:
    # -- If enabled, ServiceMonitor resources for Prometheus Operator are created
    enabled: false

Set the SelfMonitor to false

  selfMonitoring:
    enabled: false

Set the Grafana Agent to False

    grafanaAgent:
      # -- Controls whether to install the Grafana Agent Operator and its CRDs.
      # Note that helm will not install CRDs if this flag is enabled during an upgrade.
      # In that case install the CRDs manually from https://github.com/grafana/agent/tree/main/production/operator/crds
      installOperator: false

Set the Loki Canary to false

  lokiCanary:
    enabled: false

Below is the link for a configured helm chart for the LOKI Helm chart

Commands to install LOKI

kubectl create ns loki

helm install loki grafana/loki --version 5.20.0 -n loki -f loki-values.yaml

Commands to install Promtail and Grafana

helm install my-promtail grafana/promtail --version 6.15.1 -n loki

helm install my-grafana grafana/grafana --version 6.59.4 -n loki

Conclusion

The Grafana Loki has updated the architecture to 3 targets which are READ, WRITE, and BACKEND. The blog helps you to deploy the LOKI on EKS with S3 long-term storage.

In case you want to deploy the 2 Target Model for LOKI (Read and Write) you can check the blog written on the same using the below link.

Logging Setup with Grafana LOKI-2 target