Unleashing Network Observability with OpenTelemetry

8 min readFeb 5, 2024

1 February 2024
Author: Kalman Meth

Abstract

In this blog we present the recently added support of OpenTelemetry to Flowlogs-Pipeline (FLP). We show how to configure FLP to connect to an OpenTelemetry endpoint and to provide output in OpenTelemetry format. We tested the configuration using Jaeger and Instana, and share screenshots of what the output looks like in these Graphical User Interfaces (GUIs). Note that this configuration solution should be able to work beyond just these two tools of Jaeger and Instana; it is extensible enough to work with any OpenTelemetry-compatible observability tool.

What is Flowlogs-pipeline (FLP)?

Flowlogs Pipeline (a.k.a. FLP) is an observability tool that consumes logs from various inputs, transforms them, and exports logs and/or metrics to a chosen target (e.g. export logs to Loki and/or time series metrics to Prometheus). This data may then be used to perform various kinds of analyses for performance, security, accounting, alerting, or other purposes. We use the term pipeline to describe the sequence of operations performed on the data: ingest, transform, extract, encode, etc. The use of a multi-phase pipeline makes it easier to reuse specific phases while implementing new features.

Existing output environments include Prometheus, Loki, and Kafka.

FLP is an integral part of the Network Observability Operator in RedHat Openshift.

More details are available on the FLP github (https://github.com/netobserv/flowlogs-pipeline) site or on our earlier blog (https://medium.com/@meth_36197/network-insights-in-a-distributed-environment-18d5ab89f999) describing the technology.

OpenTelemetry

OpenTelemetry is an observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. OpenTelemetry is vendor- and tool-agnostic, enabling it to be used with a broad variety of observability backends. The OpenTelemetry framework includes specifications of components, protocols to define telemetry data, semantic conventions and naming schemes for data types, SDKs, instrumentation libraries, a Data Collector and Exporter, and various other tools.

FLP and OpenTelemetry

Because of the growing adoption of OpenTelemetry, it makes sense for us to provide FLP with the capability to provide OpenTelemetry-compatible flow logs, metrics, and traces.

In this blog, we discuss the OpenTelemetry features that were recently added to FLP and show how to use them. We show some screen shots of how the output looks in some tools such as Jaeger and Instana. This closely follows the example provided in https://github.com/netobserv/flowlogs-pipeline/blob/main/contrib/opentelemetry/demo.md.

Alignment with OpenTelemetry Syntax

In our setup, we use the netobserv-ebpf-agent to produce flow logs. A typical flow log entry may look like this:

{
 Bytes: 1452,
 DstAddr: 172.30.249.103,
 DstMac: 0A:58:0A:81:02:01,
 DstPort: 8080,
 Packets: 6,
 Proto: 6,
 SrcAddr: 10.129.2.16,
 SrcMac: 0A:58:0A:81:02:10,
 SrcPort: 38046,
 TimeReceived: 1705565570,
 }

The format of the flow log is a collection of <key: value> pairs. The keys of the flow-log generated by netobserv-ebpf do not match the semantics specified by OpenTelemetry Semantic Conventions. FLP allows the keys to be easily renamed via configuration parameters.

- name: otlp_transform
  transform:
    type: generic
    generic:
      policy: replace_keys
      rules:
        - input: DstAddr
          output: destination.address
        - input: DstMac
          output: destination.mac
        - input: DstPort
          output: destination.port
        - input: SrcAddr
          output: source.address
        - input: SrcMac
          output: source.mac
        - input: SrcPort
          output: source.port

In this configuration snippet, we specify that the field named ‘DstAddr’ should be changed to a field named ‘destination.address’. This is now consistent with OpenTelemetry semantics. The resulting flow log would then look like this:

{
 Bytes: 1452,
 destination.address: 172.30.249.103,
 destination.mac: 0A:58:0A:81:02:01,
 destination.port: 8080,
 Packets: 6,
 Proto: 6,
 source.address: 10.129.2.16,
 source.mac: 0A:58:0A:81:02:10,
 source.port: 38046,
 TimeReceived: 1705565570,
 }

Other fields can be similarly renamed.

Flow Log enrichment

Later in the pipeline, we specify a stage to perform enrichment of the flow log data with Kubernetes related fields. This is specified as follows:

- name: enrich
  transform:
    type: network
    network:
      rules:
        - input: source.address
          output: source.
          type: add_kubernetes
          assignee: otel
        - input: destination.address
          output: destination.
          type: add_kubernetes
          assignee: otel

The ‘add_kubernetes’ rule generates new fields based on the ‘input’ field of the rule. In our example, for a flow log that has a field named ‘source.address’ (that was previously converted from ‘SrcAddr’), additional fields are generated associated with that value. If the flow log contains a field with a key `source.address`, then it enriches that flow log entry with additional fields such as k8s.type, k8s.pod.name, k8s.service.name, etc, where the actual fields depend on the type of object it represents. This additional information is obtained directly from the kubernetes cluster on which FLP is running. The new fields’ names are prepended with the specified prefix (‘source.’ or ‘destination.’ in our example,) and are chosen in the style of OpenTelemetry. The resulting flow log may then look something like this:

{
bytes: 1452
destination.address: 172.30.249.103
destination.k8s.name: ui
destination.k8s.namespace.name: mesh-arena
destination.k8s.owner.name: ui
destination.k8s.owner.type: Service
destination.k8s.service.name: ui
destination.k8s.type: Service
destination.mac: 0A:58:0A:81:02:01
destination.port: 8080
packets: 6
protocol: 6
source.address: 10.129.2.16
source.k8s.host.ip: 10.0.168.105
source.k8s.host.name: ip-10–0–168–105.ec2.internal
source.k8s.name: stadium-base-5db66b76fd-vwlqp
source.k8s.namespace.name: mesh-arena
source.k8s.owner.name: stadium-base
source.k8s.owner.type: Deployment
source.k8s.pod.name: stadium-base-5db66b76fd-vwlqp
source.k8s.type: Pod
source.mac: 0A:58:0A:81:02:10
source.port: 38046
timereceived: 1705565570
}

The added fields all contain ‘k8s’. The specified source address is associated with a Pod from a Deployment named ‘stadium-base’ and the destination address is associated with a Service named ‘ui’.

OpenTelemetry Encoding in FLP

To send data from FLP to OpenTelemetry, we have essentially two steps: (1) connect to an OpenTelemetry endpoint and (2) send formatted data to the OpenTelemetry endpoint in the proper format. See the OTLP Protocol for more details. The full set of options to configure the FLP OpenTelemetry stages can be found in https://github.com/netobserv/flowlogs-pipeline/blob/main/docs/api.md.

Definition of an OpenTelemetry Endpoint

OpenTelemetry connections usually require at least three bits of information: target address, target port, and transport specification (either http or grpc). In addition, authorization information must usually be provided, either using TLS or passing authorization information in message headers. The configuration of an OpenTelemetry encode stage for FLP, specifying these parameters, would look something like this.

- name: otel
  encode:
    type: otlplogs
    otlplogs:
      address: my-otelcol-collector-headless.otlp.svc
      port: 4317
      connectionType: grpc
      tls:
        insecureSkipVerify: true

Additional parameters may be specified for TLS as in other FLP stages (see https://github.com/netobserv/flowlogs-pipeline/blob/main/pkg/api/tls.go).

Logs

FLP already naturally processes flow logs. The standard structure of a flow log is a set of <key: value> pairs, as described above. These are provided to an OpenTelemetry endpoint in the form of a json string and as a set of OpenTelemetry attributes (<key: value> pairs).

Metrics

Similar to Prometheus, OpenTelemetry provides support for three types of metrics: counters, gauges, and histograms. The syntax to specify OpenTelemetry metrics for FLP is the same as to specify Prometheus metrics for FLP (see https://github.com/netobserv/flowlogs-pipeline/blob/main/pkg/api/encode_prom.go).

Traces

FLP naturally produces logs and metrics of network traffic. Traces in the classical sense are generally produced inside an application using spans (see https://opentelemetry.io/docs/concepts/observability-primer/#spans), and FLP has no way to reproduce the trace information of an application. However, FLP can provide some insights into the network traffic relative to the application based on the flow logs, which contain both ‘source’ and ‘destination’ information. A typical flow log may contain both the source and the destination IP address/port. Each address/port is typically associated with a particular application. In our trace module, we separate out the fields in a flow log that relate to the source and/or to the destination, and allow them to be reported separately as attributes (tags) of sub-spans, similar to what can be done in a regular trace.

The following screen shots show in the Jaeger GUI how an FLP OpenTelemetry trace is depicted. We see a parent span that emanates from ‘flp_enode’ (this is the name of the module that produced the trace), and we see two subspans labelled ‘source.’ and ‘destination.’.

These labels (‘source.’ and ‘destination.’) are actually the prefixes that we inserted earlier in our enrichment stages. We specify in our configuration that FLP should create a subspan for each of these specified prefixes.

- name: otel
  encode:
    type: otlptraces
    otlptraces:
      address: my-otelcol-collector-headless.otlp.svc
      port: 4317
      connectionType: grpc
      tls:
        insecureSkipVerify: true
      spanSplitter:
        - source.
        - destination.

In the subspans, the prefix is removed, and we are left with fields that match the OpenTelemetry semantic conventions. This may allow a smart application to cross reference data provided in these traces with other available data associated with the same OpenTelemetry field values.

Additionally, we show a similar example with the IBM Instana GUI. In the figure below we see the parent span labelled ‘flp_encode’ and two subspans labelled ‘source.’ and ‘destination.’. We see on the right side the tags that are the fields of the flow log. Here also we see that in the parent span we have the original key with the ‘source.’ prefix and in the subspan the prefix has been removed, leaving us with field names that are consistent with the OpenTelemetry naming conventions.

We show here the same information presented as a flow log in the Instana GUI. This is achieved by sending the raw flow logs instead of traces.

- name: otellogs
  encode:
    type: otlplogs
    otlplogs:
      address: my-otelcol-collector-headless.otlp.svc
      port: 4317
      connectionType: grpc
      tls:
        insecureSkipVerify: true

We see the fields destination.k8s.pod.name and source.k8s.pod.name in the log entry, similar to the way that they appeared in the tags of the trace entry.

Summary

To summarize, we discussed the recently added support of OpenTelemetry to the Flowlogs Pipeline (FLP). FLP can now export flow data (logs, metrics, and traces) using OTLP that conforms to OpenTelemetry semantic conventions. We showed how to configure FLP to connect to an OpenTelemetry endpoint and how to provide output in OpenTelemetry format. We showed several stages of the process to transform a flow log into the desired format. We also showed some screen shots of what the output looks like in a GUI.