Integration of Thanos with Prometheus and S3 as Storage

An open-source project called Thanos offers a scalable and highly accessible way to store and query metrics data, thereby smoothly expanding Prometheus’s capabilities. The procedures to configure Thanos in a Kubernetes cluster with the Kube-Prometheus-Stack(Helm Chart) and metrics data stored in an S3 bucket are described in this blog. You can accomplish high availability and effective long-term storage, as well as guarantee the dependability and longevity of your Prometheus metrics data, by putting Thanos into practice.

Why Thanos?

Within the Kubernetes community, Prometheus is a popular toolset for monitoring and alerting, although its scalability and long-term storage are limited. In order to overcome these obstacles, Thanos provides the following essential features: Global Query View: You can compile and query metrics data from several clusters using Thanos’ single global query view, which is available to all Prometheus instances. High Availability: By facilitating the implementation of highly available Prometheus configurations, Thanos lowers the possibility of data loss and guarantees ongoing monitoring even in the event that a single Prometheus server fails. Long-Term Storage in Object Storage: You may use Thanos to store Prometheus metrics data in object storage services like Amazon S3, Google Cloud Storage, or Azure Storage, which are scalable and reasonably priced. Compaction and Deduplication: Thanos

Part-1 Configuring Thanos Side-car with Kube-Prometheus-Stack:

Step 1: Adding Helm Repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Step 2: Installing the Kube-Prometheus-Stack chart:

helm install prometheus  --create-namespace -n prometheus prometheus-community/kube-prometheus-stack --version 55.8.1

Step 3: Edit the “values.yaml” file to include Thanos Side-car configuration:

prometheus:
  prometheusSpec:
    disableCompaction: true
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 1Gi
    thanos:
      image: quay.io/thanos/thanos:v0.28.1
      objectStorageConfig:
        existingSecret:
          name: thanos-objstore-config
          key: thanos.yaml

  thanosService:
    enabled: true
    annotations: {}
    labels: {}
    externalTrafficPolicy: Cluster
    type: ClusterIP
    portName: grpc
    port: 10901
    targetPort: "grpc"
    httpPortName: http
    httpPort: 10902
    targetHttpPort: "http"
    clusterIP: ""
    nodePort: 30901
    httpNodePort: 30902

Step 4: Upgrading the installed chart with the above “values.yaml”:

helm upgrade --install prometheus  --create-namespace -n prometheus prometheus-community/kube-prometheus-stack --version 55.8.1 --values values.yaml

Step 5: Create a file named “thanos-storage-config.yaml” for S3 Storage configuration:

type: s3
config:
  bucket: thanos-store #S3 Bucket Name
  endpoint: s3.<region>.amazonaws.com #S3 Regional endpoint
  access_key: <aws-account-id>
  secret_key: <aws-account-secret>

Step 6: Create secret using below command:

kubectl -n prometheus create secret generic thanos-objstore-config --from-file=thanos.yaml=thanos-storage-config.yaml

Part-2 Thanos Components installation and configuration:

Here we focus on how to deploy and configure Thanos Components which are responsible to collect all the data from the Prometheus and S3 Storage that we deployed in the first part. We will be using “kube-thanos” manifests here for configuring the Thanos Components. You can choose to clone kube-thanos repository and use the manifest folder.

git clone git@github.com:thanos-io/kube-thanos.git

Step 1: Create namespace for Thanos Components:

kubectl create namespace thanos

Step 2: Configuring “thanos-query-deployment.yaml”:

To enable the communication between thanos-query-deployment.yaml to the thanos-sidecars we need to add this:

- --store=dnssrv+_grpc._tcp.thanos-prometheus-<cluster_name>.<domain_name>:10901

into the args section with Thanos sidecar GRPC endpoint.

 - args:
   - query
   - --grpc-address=0.0.0.0:10901
   - --http-address=0.0.0.0:9090
   - --log.level=info
   - --log.format=logfmt
   - --query.replica-label=prometheus_replica
   - --query.replica-label=rule_replica
   - --endpoint=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local:10901
   - --endpoint=dnssrv+_grpc._tcp.thanos-receive-ingestor-default.thanos.svc.cluster.local:10901
   - --store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local:10901
   - --store=dnssrv+_grpc._tcp.prometheus-kube-prometheus-thanos-discovery.prometheus.svc.cluster.local:10901
   - --query.auto-downsampling

Step 3: Create secret in “thanos” namespace with similar configuration file used for above secret to configure Thanos-Store:

kubectl -n thanos create secret generic thanos-objectstorage --from-file=thanos.yaml=thanos-storage-config.yaml

Step 4: Deploying Thanos manifests files:

kubectl apply -f manifests -n thanos

Conclusion:

By following these steps, you have successfully Integrated Thanos with Prometheus and S3 as Storage. This setup ensures the reliability, scalability, and long-term storage of your metrics data, providing a robust monitoring solution for your Kubernetes environment. Share this documentation with your friends/colleauges to enable them to implement Thanos in their environments.

Happy Learning! :)