OpenTelemetry Operator: Decreasing Your Otel Adoption Time Through Auto-Instrumentation and More

If you are in the process of migrating your observability platform to OpenTelemetry and are concerned about the adoption time and effort, you will find this article interesting.

Observability according to Midjourney AI.

The biggest concern so many observability engineers have when it comes to adopting OpenTelemetry is the adoption time and effort. Teams need time to update their frameworks and accept the new changes. Depending on the size of the company and the complexity of the architecture, this adoption can take months. But what if I told you that it is possible to offer out-of-the-box instrumentation for OpenTelemetry, with minimal effort from the engineers in your company?

The Otel community has provided us with the amazing OpenTelemetry Operator, which amongst many other features, provides auto-instrumentation capabilities for several technologies. All the information I will provide below is already documented and I am hoping that this article will serve you as a knowledge map.

What is the OpenTelemetry Operator?

It is an operator that manages:

1. A collector instance, which has the same abilities as a standalone Otel collector. You can deploy it as “deployment”, “statefulset”, “daemonset”. You are able to use the same receivers, exporters, processors and configurations as a standalone collector.
2. An auto-instrumentation instance, which provides auto-instrumentation capabilities by injecting Opentelemetry instrumentation libraries during application runtime. You could use the SDKs provided by the community or your custom ones.

It is based on the Kubernetes Operator.

How to implement it?

First, you can find the charts here. It is necessary to use a certificate to deploy the Operator. You can use certmanager, and as of the latest version, you can use self signed certificates.

Second, make sure to have the templates for: the operator collector and the auto-instrumentation object.
If you are using helm, they can look as below:

#Sample helm template for the auto-instrumentation object
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: infrastructure-opentelemetry-instrumentation
spec:
  {{- if .Values.instrumentation.exporter }}
  exporter:
    endpoint: {{ .Values.instrumentation.exporter.endpoint }}
  {{- end }}
  {{- if .Values.instrumentation.propagators }}
  propagators:
    {{- range .Values.instrumentation.propagators }}
    - {{ . }}
    {{- end }}
  {{- end }}
  {{- if .Values.instrumentation.sampler }}
  sampler:
    {{- toYaml .Values.instrumentation.sampler | nindent 4 }}
  {{- end }}
{{- with .Values.instrumentation.extraValues }}
  {{- toYaml . | nindent 2 }}
{{- end }}

#Sample template for the collector. 
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: infrastructure-opentelemetry-operatorcollector
spec:
  mode: {{ .Values.mode }} # could be "deployment", "statefulset", "daemonset"
  {{- if .Values.serviceAccount.enabled }}
  serviceAccount: {{ .Values.serviceAccount.name }}
  {{- end }}
  {{- with .Values.config }}
  config: |
    {{- toYaml . | nindent 4 -}}
  {{- end }}
  {{- with .Values.additionalValues -}}
  {{- toYaml . | nindent 2 -}}
  {{- end }}

Third, you need to make sure the collector that is being managed by the Operator, is prepared to receive and export telemetry signals. For this, it is important to understand, that auto-instrumented applications will send OTLP format metrics and traces, so you need to:

1. Make sure the otlp receiver is configured :

collector:
  config:  
    receivers:
        otlp:
          protocols:
            grpc:
              endpoint: ${env:MY_POD_IP}:4317
            http:
              endpoint: ${env:MY_POD_IP}:4318

2. Make sure the otlp receiver is added to the pipelines

collector:
  config:
    pipelines:
          metrics:
            exporters:
              - logging
            processors:
              - memory_limiter
              - batch
            receivers:
              - otlp #configured as needed
          traces:
            exporters:
              - logging
            processors:
              - memory_limiter
              - batch
            receivers:
              - otlp #configured as needed
              - jaeger
              - zipkin

Next, add the correct configurations for the auto-instrumentation

instrumentation:
  sampler:
    type: parentbased_traceidratio
    argument: "0.2" #sampling 10% on the instrumentation side
  extraValues:
    go:
      image: #add your custom image, otherwise the default image will be used
    java:
      image:
    dotnet:
      image:
    nodejs:
      image:
    python:
      image:
  

You need to keep in mind, that the sampler configured above samples on the instrumentation side, what the application will send to the collector. So if you add another sampling processor in the collector, like tail sampling or probabilistic sampling, they will work independently of each other. You will sample your telemetry signals twice.

You will be able to find information about the technologies that currently are supported for auto-instrumentation in the documents below:

Java: https://github.com/open-telemetry/opentelemetry-java-instrumentation

.NET: https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation

And your OpenTelemetry Operator is ready. Simple as that.

How to inject auto-instrumentation into your applications?

It’s as easy as adding a simple annotation to your application. You can find all this information here.
Once you have added this annotation to your application, your application will boot and you will see a new init container called “ OPENTELEMETRY AUTO-INSTRUMENTATION” in your pods and telemetry signals should start flowing.

#injecting java auto-instrumentation
instrumentation.opentelemetry.io/inject-java: "true"

#injecting go auto-instrumentation
instrumentation.opentelemetry.io/inject-go: "true"
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path/to/container/executable"

And just like that, you have provided your engineers with an instrumentation tool that is super easy to adopt, and that provides all the information they need to implement observability on their applications. And if you feel like the libraries provided by the community do not fit all your needs, you can always contribute to extending them or use your own in-house libraries.

I hope this article has inspired you to give it a try to the OpenTelemetry Operator, to take advantage of all it offers. My journey with Opentelemetry migration and implementation continues, and every day I discover new features that I love to share with you. I have been experimenting with several different processors for custom attributes and labels, so stay tuned for the next article!

And if you are interested to hear more about my passion and work with Observability and OpenTelemetry, you can watch/listen to some of my appearances:

1. On-Call me maybe podcast episode
2. Otel in Practice- Observability is a team sport
3. OpenTelemetry Q&A

Further Reading:

OpenTelemetry Tracing: Everything you need to know

OpenTelemetry: A full guide

Combining OTel and Prometheus metrics for alerting machine