Unveiling the black box with observability stack

Deploying open source Observability Stack for application insights: Promtail, OpenTelemetry, Prometheus, Loki, Tempo & Grafana

Ever feel like your application is a total mystery? It works most of the time, but when something goes wrong, you’re left staring at a screen like, “Okay, what did you just do?”

Yeah, that’s the struggle with black box applications. They get the job done, but what’s happening behind the scenes is anyone’s guess. This can make troubleshooting issues a nightmare, leaving you frustrated and wasting precious time even if we assume familiarity with the service architecture and data flow.

But fret not as there’s a way to see what’s really going on! Enter the observability stack, your secret weapon for understanding your application. Think of it like a super cool toolbox for peeking inside and figuring out what’s making your app tick. Imagine having superpowers to see error messages (thanks Promtail!), track how your app performs (hello Prometheus!), and even rewind time to see exactly what happened before a crash (thanks Tempo!). Pretty awesome, right?

With the observability stack by your side, you can go from black box blues to application transparency. No more guessing games, just clear insights to help you fix problems fast and keep your app (FastAPI microservices) running smoothly. In the next section, we’ll break down the tools in this amazing stack and show you how they work together to bring your app into the light!

What is the observability stack?

The observability stack is a collection of tools that work together to gather, analyze, and visualise data about your application’s performance and health. This is done by collecting signals such as traces, logs and metrics. More importantly, in troubleshooting distributed systems, logs tells us what happened and traces tells us where and when it happened.

Logs, akin to detailed records, provide a chronological account of events within the application. However, to gain a holistic understanding of these events, supplementing it with trace information is crucial. Traces, comparable to timestamps and references, link individual log entries to the broader trace context. Each service has its own span which are connected to a parent trace. By connecting these spans, the entire trace across the multiple services can be constructed. To achieve this, connective information (context) consisting of the span ID, parent ID and the trace ID has to be propagated and updated along the services.

The linkage between logs and traces empowers pinpointing the precise location and sequence of events within a distributed system, which can be very effective for troubleshooting purposes. For example if a 404 error is raised in the logs, the trace can be use to narrow the debugging efforts to a specific service.

Collection of signals using observability stack on Kubernetes

Open Telemetry Instrumentation

The process starts with instrumenting the microservices in the backend running on kubernetes (on your application namespace) using OpenTelemetry libraries. OpenTelemetry is a vendor-neutral instrumentation framework that allows applications to generate telemetry data in a common format. This ensures that regardless of the programming language or framework used, data can be collected and understood by the observability stack.

There are generally 3 types of instrumentation arsenal to choose from, namely automatic, manual and programmatic instrumentation (combination of both automatic and manual). As you can already guess, automatic instrumentation will require the least extent of code refactoring and conversely manual instrumentation will require significantly more code refactoring although it also brings about greater flexibility and finer details in trace visualisation subsequently.

If you find yourself in a situation like me where there are countless number of microservices communicating via RESTful APIs, it might be a good idea to explore automatic instrumentation. Fear not! The implementation is pretty simple and sweet. To find out more in detail you can click here for reference.

Assuming you are running your application using Dockerfiles, you can include the following lines in your Dockerfile before the uvicorn executable command. opentelemetry-bootstrap tool installs the required instrumentation dependencies depending on your requirements.txt which is required for the instrumentation agent to work.

RUN opentelemetry-bootstrap -a install

ENTRYPOINT ["opentelemetry-instrument"]

CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "your port",  "--log-config", "logging_config.yaml"]

Additionally, the OpenTelemetryMiddleware package needs to be imported from the opentelemetry.instrumentation.asgi library which acts as a wrapper for the FastAPI application that catches incoming request and outgoing responses to capture the context for trace spans.

While automatic instrumentation simplifies the process by abstracting away the need for code changes (refactoring), this very convenience can lead to unexpected problems. One such problem faced during my implementation was the flooding of the logs due to the instrumentation of packages such as uvicorn and urllib3 which might not be useful. To mitigate this, you can set the logger info level to DEBUG or lower and then set the root level to INFO to filter out the unwanted logs in the logging configuration.

Remember the benefits of interlinking the logs and traces for more effective debugging? This is where the magic comes to play. In the same logging_config.yaml, the logs format can be specified to include both the trace ID (otelTraceID) and span ID (otelSpanID). Feel free to adopt other standard formatting if preferred.

formatters:
  standard:
    format: "%(asctime)s - %(levelname)s - %(name)s - %(message)s"
  access:
    format: >-
      %(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d]
      [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s]
      - %(message)s

Promtail

Configured as a daemon set in the configurations.yaml, Promtail will scrape logs from the kubernetes pods, tagging the logs with labels followed by exporting the logs to OpenTelemetry collector. This target label will be used for aggregating and filtering logs subsequently during visualisation in Grafana. For example in the code snippet of the configmap for Promtail, these are actions that extract the values from the existing __meta_kubernetes_namespace and __meta_kubernetes_pod_name labels and assign them to new labels named namespace, pod or containers. Again feel free to give your own labels to facilitate your workflow.

source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container

When deploying Promtail on kubernetes, the components of the role based access specifically cluster role, service accounts and role binding yaml files will need to be configured according to your application settings. The daemon set yaml configuration which defines how Promtail pods are deployed across the cluster can then specify the service account that was configured previously with the relevant access required. These permissions are necessary for Promtail to discover kubernetes pods that it should scrape logs from and access relevant service information for communication.

In order for Promtail to export the logs to Open Telemetry Collector Loki endpoint, the client url will have to specified in the configmap.yaml as shown in the example code snippet:

data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: http://otel-collector:3100/loki/api/v1/push
      tenant_id: 1

Data Propagation to data source

OpenTelemetry Collector

Apart from receiving logs from Promtail, the OpenTelemetry Collector can also collect traces and metrics from other sources that use OpenTelemetry. For instance, the trace from the OpenTelemetry instrumentation can be ingested by specifying the endpoints for traces and metrics as part of the environment variable in the deployment.yaml file for the microservices and the receiver in the configmap.yaml for Open Telemetry Collector.

env:
  - name: OTEL_SERVICE_NAME
  value: {{ .Values.appName }}
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
  value: {{ .Values.global.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT }}
  - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
  value: {{ .Values.global.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL }}
  - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
  value: {{ .Values.global.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT }}
  - name: OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
  value: {{ .Values.global.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL }}

otel-collector-config: |
    receivers:
      otlp:
        protocols:
          http:
            endpoint: "0.0.0.0:4318" # Example default port for http
      loki:
        protocols:
          http:
            endpoint: "0.0.0.0:3100" # Example default port for Loki

The Collector can process, transform, and route the telemetry data to different backends for storage and analysis. As a central hub for the logs, traces and metrics, Open Telemetry Collector can then export these signals to Loki, Tempo and Prometheus respectively. This can be achieved by indicating each endpoints in the exporters section of the otel-collector-config.yaml similar to the example here:

exporters:
      loki:
        endpoint: http://loki:3100/loki/api/v1/push
        headers:
          "X-Scope-OrgID": "1"
      otlp:
        endpoint: tempo:4317 # default otlp port is 4317
        tls:
          insecure: true
      otlphttp:
        endpoint: http://tempo:port # your exposed port for http server for tempo
      prometheus:
        endpoint: 0.0.0.0:9090

Loki, Tempo and Prometheus

Depending on your preference, either a helm chart or direct deployment can be used to spin up these data source services on your kubernetes cluster on your observability stack namespace. For more step by step instruction for helm chart deployment (Loki) click here. Similarly for Tempo and Prometheus, the documentation can be found in Grafana Labs documentation. Ensure that the ports exposed in the containerPort is the same as the endpoint specified in the OpenTelemetry Collector.

As for volumes used for storing ingested data, you can consider using object storage systems like MinIO or Amazon S3 buckets. Additionally, you can also leverage the underlying storage infrastructure associated with EKS, such as Amazon EBS for block storage or Amazon EFS for shared file storage.

Data visualization

Grafana

Grafana is a visualisation tool that allows you to query and create dashboards from the data stored in Prometheus, Loki, and Tempo. These dashboards can display metrics, logs, and traces in a variety of formats, such as graphs, tables, and charts. This enables you to gain insights into the health and performance of your application.

Since Grafana uses Loki, Tempo and Prometheus as data sources, it will have to be specified under the datasource.yaml within the configmap.yaml. For instance if I want to use Loki and Tempo as the data sources to visualise and query the logs and traces in the Grafana UI, I can specify the data sources as shown:

data:
  datasource.yaml: |
    apiVersion: 1
    datasources:
    - name: Loki (auto)
      type: loki
      access: proxy
      url: http://loki:3100
      jsonData:
        httpHeaderName1: "X-Scope-OrgID"
      secureJsonData:
        httpHeaderValue1: '1'
    - name: Tempo (auto)
      type: tempo
      access: proxy
      url: http://tempo:3200
      jsonData:
        httpHeaderName1: "X-Scope-OrgID"
      secureJsonData:
        httpHeaderValue1: '1'

Since you’ll be accessing the Grafana interface from outside your local network, we need to configure it for external access. To achieve this, we’ll use a NodePort service type. This service type exposes the application on a random port on the Kubernetes cluster and forwards traffic to the default Grafana port (3000) running on the pods. This way, you can access Grafana externally using the assigned NodePort.

Generating insights from collected signal

Finally after all the configurations are out of the way, we are now able to query the logs and traces using the Grafana UI such as filtering all logs that have status code 401 and those which contains certain keywords (e.g migration). The label filters which we have added in Promtail can also be chosen under the label filters drop down bar (namespace selected in my case).

The logs shown after you click the run query should include the timestamp, traceID and spanID as specified in your logging format previously. This traceID can then be used to query Tempo to see the entire trace of the erronous API call for deep dive.

Other than querying for the specific traceID or spanID, you can also identify latencies across your API calls by indicating the duration threshold or using the other search tags as preferred.

Its a wrap

Phew, we made it to the end! High fives for sticking with it!

To summarise, we’ve successfully set up the observability stack on Kubernetes in a separate namespace from the application stack. We’ve also managed to propagate the instrumented data all the way to Grafana, where we can now visualise and understand the workings of our complex application.

Hopefully this has helped you to achieve some transparency in your black box and make your debugging life easier! Till next time!

P.S I would like to express my sincere gratitude to everyone on the Sensei Team at GovTech for providing me with the opportunity to explore and work on meaningful projects during my internship. I am immensely thankful for the support and guidance I received throughout this period, which has been invaluable to my learning and professional development.