Grafana Loki multi tenant setup in EKS clusters

Grafana Loki promtail multi tenant setup in EKS Clusters

In this article, I’ll show you how to create a robust, scalable logging system for enterprise-grade applications using Loki, Grafana and Promtail in EKS clusters. This logging setup will allow you to:

a) Install the full Loki stack in one AWS management account with persistent storage (AWS S3 for Loki & EBS volume for Grafana)
b) Run Promtail in EKS tenant clusters configured in other AWS accounts, scraping logs from a sample application and reporting them to the central Loki instance in the management account.
c) Implement RBAC in Grafana to allow tenants to only see their logs when they log into Grafana, while allowing administrators to see entire tenant logs.

Prerequisites

• The AWS CLI configured (run aws configure).
• kubectl and eksctl installed.

Setting up the management EKS cluster

a) Create the cluster

#eksctl create cluster — name loki-promtail — region us-east-1 — managed

It would take around 15 minutes to complete. You should have kubectl context configured to communicate with your newly created cluster.

Note: eksctl supports specifying multiple instance types for managed and self-managed nodegroups Please refer https://eksctl.io/usage/instance-selector/ for more details on it.

b) Install the AWS Load Balancer Controller add-on

When you create a Kubernetes Service of type LoadBalancer, the AWS cloud provider load balancer controller creates AWS Classic Load Balancers by default, but can also create AWS Network Load Balancers. The AWS Load Balancer Controller creates AWS Network Load Balancers, but doesn’t create AWS Classic Load Balancers.
Ref: https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html

c) Install aws-ebs-csi-driver

Here we use EBS volume for Grafana persistent storage. EBS Container Storage Interface (CSI) driver allows EKS clusters to manage the lifecycle of Amazon EBS volumes for persistent volumes.
Reference: https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html

d) Install Grafana & loki as separate services using helm

#helm repo add loki https://grafana.github.io/loki/charts

#helm install loki — namespace=monitoring — set grafana.enabled=false,promtail.enabled=true grafana/loki-stack — values loki-values.yaml

#helm install grafana grafana/grafana — set persistence.storageClassName=”gp2" — set persistence.enabled=true — set service.type=LoadBalancer — namespace=monitoring

Loki, Grafana and Promtail can all be installed together using the loki-stack chart. I’ve installed Grafana and Loki as separate deployments here for ease of management.

loki:
  memberlist:
    abort_if_cluster_join_fails: false
  limits_config:
    reject_old_samples: false
  config:
    schema_config:
      configs:
        - from: 2022-10-28
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            prefix: loki_
            period: 24h
    storage_config:
      aws:
        s3: s3://us-east-1/loki-data
        access_key_id: <AWS_ACCESS_KEY_ID> 
        secret_access_key: <AWS_SECRET_ACCESS_KEY>
        s3forcepathstyle: true
        bucketnames: loki-data
        region: us-east-1
        insecure: false
        sse_encryption: false
        http_config:
          idle_conn_timeout: 90s
          response_header_timeout: 0s
          insecure_skip_verify: false
    compactor:
      working_directory: /data/compactor
      shared_store: s3
      compaction_interval: 5m
    auth_enabled: true
    chunk_store_config:
      max_look_back_period: 0s

e) Now edit the loki service and change type to LoadBalancer. I’ve used the following annotations in the loki service to use an AWS Network load balancer:

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip

f) Share NLB with other AWS accounts through AWS PrivateLink
Reference: https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html

Setting up the tenant EKS cluster:

a) Run the following command as the other aws account:

#eksctl create cluster — name Tenant1 — region us-east-1 — managed

b) Add a VPC endpoint to the NLB endpoint service configured in the management EKS Cluster.

c) Install Promtail agent:

#helm upgrade promtail loki/promtail -n promtail -f promtail-tenant1-values.yaml

loki url IP needs to be the tenant endpoint private dns IP. Private DNS can be obtained from the tenant vpc endpoint.

Tenant auth details are defined in promtail-tenant1-values.yaml. Here’s a sample yaml file.

config:
 clients:
    - url: http://10.0.7.142:3100/loki/api/v1/push
      tenant_id: tenant1
      basic_auth:
        username: tenant1
        password: secret

Grafana Setup:

a) Login Grafana as admin user.
b) Create Grafana Organizations and Add Loki data source (url is http://loki:3100 in this setup). Also enable Basic Auth and enter the credentials defined in promtail-tenant1-values.yaml
c) Enter X-Scope-OrgID value (defined in promtail-tenant1-values.yaml) in Custom HTTP Headers
d) Save & Test

Thanks again for reading my article. Hope you have liked it.

Until next time…….