So You Built A Custom Collector with the OpenTelemetry Collector Builder…Now What?

Things nobody tells you when you build your own OTel Collector distribution

Water tower at dusk.

There are plenty of blog posts out there that explain how to build your own OpenTelemetry (OTel) Collector distribution, and you can even find the steps in the official OpenTelemetry documentation.

Unfortunately, when I went to build my own Collector distribution, I was left with a few blanks to fill on my own. Today, I’ll discuss to do it myself, I ran into a few snags that we’re covered in any of the places I looked. (Maybe I didn’t look hard enough? 🤷‍♀️) So today, I will share with you some of the snags that I hit when I attempted to build my own Collector distribution, and what I learned along the way.

Building a Collector distribution

To build an OpenTelemetry Collector distribution, you need to install the OpenTelemetry Collector Builder (OCB) tool. I followed the instructions in the OTel docs, and it worked pretty well.

My goal is to eventually run my Collector image in Kubernetes, so when I build my Collector image, I want to be able to build it in both linux/arm64 and linux/amd64 architectures, so I started with this:

if [ $(uname -m) = x86_64 ]; then
    DISTRO="amd64"
elif [ $(uname -m) = aarch64 ]; then
    DISTRO="arm64"
fi

echo "Distro is ${DISTRO}"

curl --proto '=https' --tlsv1.2 -fL -o ocb \
https://github.com/open-telemetry/opentelemetry-collector/releases/download/cmd%2Fbuilder%2Fv0.102.1/ocb_0.102.1_linux_${DISTRO}
chmod +x ocb
sudo mv ocb /usr/local/bin/ocb

Gotcha #1

Take note of the version of the OCB. I wanted to build a distribution based on version 0.102.1 of the Collector, so I had to make sure that the binary I pulled from GitHub was version 0.102.1

Gotcha #2

Oh, by the way, you also need to make sure that you install Go on your machine, because the OCB tool uses Go to build the Collector distribution. I was using a Development (Dev) Container for my work, and guess what I didn’t have installed in my Dev Container? You guessed it: Go!

Gotcha #3

The OCB builds a Collector distribution based on a YAML configuration file which lists the Collector components (i.e. extensions, connectors, receivers, processors, and exporters) that you wish to include.

Per the docs, I created a file called builder-config.yaml, and included the components that I needed. Mine looks like this:

# Go modules for Collector core: https://pkg.go.dev/go.opentelemetry.io/collector
# Go modules for Collector contrib: https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib

dist:
  name: otelcol-kepler-benchmark
  description: OTel Collector benchmark for Kepler
  output_path: ./_build/col


receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.102.1
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.102.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sobjectsreceiver v0.102.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver v0.102.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver v0.102.0

processors:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor v0.102.0
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.102.1
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.102.1
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.102.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.102.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.102.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.102.1
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.102.1
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.102.1

If you look closely at my file, you’ll notice that I use a combination of components from the Core, and Contrib Collector distributions.

You’ll notice that the components from the Core distribution come from the following Go package:

go.opentelemetry.io/collector/exporter/otlpexporter

The Contrib components, however, come from a different Go package:

github.com/open-telemetry/opentelemetry-collector-contrib

Yeah. That threw me for a bit of a loop. Also, remember how I’m building based on v0.102.1 of the Collector? Well, v0.102.1 is available for the Core components. Not so for the Contrib components. I had to use v0.102.0, because that was the closest version to what I needed.

How did I find this? I have enough Go knowledge to be dangerous, and I started rooting around in the OTel Go packages. This one for Core, and this one for Contrib. Be sure to bookmark these, because they’re handy. You can find the component names and versions.

💡 REFRESHER: The Core distribution is a bare-bones distribution of the Collector, and contains a base set of extensions, connectors, receivers, processors, and exporters. The Contrib distribution extends the Core distribution, and includes components created by third-parties (including vendors and individual community members), that are useful to the OpenTelemetry community at large.

I built it…now what?

Cool. I built my Collector distribution. Yayyyyy! It created a Collector binary for me. Double-yayyyyy! But…um…I need to run this thing in Kubernetes. Which means that it needs to be containerized. And guess what? I couldn’t find any documentation on how to build a Collector container.

But I did find the Collector Dockerfile in the opentelemetry-collector-contrib repository on GitHub:

FROM alpine:latest AS prep
RUN apk --update add ca-certificates

FROM scratch

ARG USER_UID=10001
ARG USER_GID=10001
USER ${USER_UID}:${USER_GID}

COPY --from=prep /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY otelcontribcol /
EXPOSE 4317 55680 55679
ENTRYPOINT ["/otelcontribcol"]
CMD ["--config", "/etc/otel/config.yaml"]

This Dockerfile copies the OpenTelemetry Collector binary that I built with the OCB into the Dockerfile. Short and sweet.

Gotcha #4

Uh…but I built my Collector distribution on an arm64 machine. I need to do a multi-arch Docker build, so that I can run this both locally and also on Kubernetes. My Kubernetes nodes are amd64, so I need to build a Docker image that is for arm64 WITH a Go binary built for arm64 architectures as well. Awwww, CRAP. 💩

LUCKILY! My former colleague and friend, Jacob Aronoff, who is one of the maintainers of the OpenTelemetry Operator, came to my rescue with this Dockerfile:

Here, we build the Collector distribution INSIDE the Dockerfile, so the Go binary gets built using the correct architecture. No fuss, no muss.

I won’t pretend to understand all the fancy Dockerfile stuff going on, but I can admire the fact that the resulting image is as minimalistic as possible, because it uses multi-stage Docker builds. Very elegant!

Gotcha #5

Unfortunately, it didn’t work out of the box, because of how I configured my builder-config.yaml file. My builder produces a binary called otelcol-kepler-bechmark inside a folder called _build/col.

So I altered the file and ended up with this as my final Dockerfile:

What’s different?

• Updated lines 8 and 9 (original file): My builder config file was called builder-config.yaml, not manifest.yaml, and my Collector config file is called otelcol-config.yaml, not config.yaml, so I updated the names accordingly.
• Removed lines 9, 10, 11 (original file): Jacob said that I probably didn’t need those lines, and he was right!
• Added line 12. I don’t know why, but Docker was complaining that ./_build/col didn’t exist, so I created that directory, and that made Docker happy.
• Added some more ports on line 30. The Dockerfile from opentelemetry-collector-contrib also included ports 55680 and 55679, so I threw them in for good measure.
• Fixed the entrypoint on line 25: After a lot of Dockerfile debugging (i.e. me logging into the container instance and rooting around), I realized that my binary was actually located at /otelcol/otelcol-kepler-benchmark, and not at /otelcol, so I fixed that.
• Removed line 15 (original file): That validation step was causing me issues, so I removed it. In hindsight, it’s probably because the path to the validator was wrong. I have a feeling that the line should be:

RUN --mount=type=cache,target=/root/.cache/go-build ./_build/col/otelcol-kepler-benchmark validate --config config.yaml

I haven’t tried it. It’s 2:15am on Pi Day (aka the best day of the year, because PIE IS AWESOME) as I write this and I’m sleepy. But I also wanted to do this brain dump ASAP before I forgot all the cool stuff I learned. Anyway, feel free to give it a whirl and see if it works.

Built, at last!

Finally, finally, I was able to build my Dockerfile. Since I needed to do a multi-architecture build, I did it like this:

GH_TOKEN=<your_github_token>
GH_USERNAME=<your_github_username>

echo $GH_TOKEN | docker login ghcr.io -u $GH_USERNAME --password-stdin
cd src/ocb

# Enable Docker multi-arch builds
docker run -it --rm --privileged tonistiigi/binfmt --install all
docker buildx create --name mybuilder --use

# Build Docker file for linux/arm64 and linux/amd64
docker buildx build --push \
  -t ghcr.io/${GH_USERNAME}/otelcol-kepler-benchmark-0.102.1:0.1.0 \
  --platform=linux/arm64,linux/amd64 .

The last line of this script will perform a multi-architecture build of the Dockerfile, and then will push it to GitHub Container Registry. If you use a different Container registry you’ll need to alter it accordingly.

Also, when you reference this image in your Kubernetes manifest, make sure that your registry either public, or if you’re using a private registry, make sure that you do the appropriate setup in Kubernetes (sorry, that’s out of scope for this blog post).

In my case, I’m deploying my OpenTelemetry Collector via the OpenTelemetry Operator, so I’m sticking the image name in my OpenTelemetryCollector resource. You can check out OpenTelemetryCollector YAML. The Collector image is line 15. It’s a long-ass file, so I’m not embedding the whole file.

💡 Want to learn more about the OTel Operator? Check out my blog series on the OTel Operator:

Adriana Villela

OpenTelemetry Operator

View list

8 stories

Gotcha #6

I initially had trouble (translation: Adriana spent 3 hours banging her head against the wall) getting the multi-architecture build working on my work M3 Mac running Podman, so I finally relented and ran the build on my personal M1 Mac running Docker Desktop with no issues.

A few days later, I updated my Podman and also cleaned up my Podman volumes, and “micraculously” (translation: Adriana should’ve done this sooner but was too lazy to do it), my multi-architecture build worked.

Moral of the story, update your software and clean up your damned volumes on your containerization tool.

Did the build actually work?

You probably want to test your newly-built Docker image (always a good idea). In which case I would suggest that you build the image and output it to Docker (so you can run it locally). And probably before you push it to your container registry.

# Build image without pushing to your container registry
docker buildx build --no-cache --load \
  -t ghcr.io/${GH_USERNAME}/otelcol-kepler-benchmark-0.102.1:0.1.0 \
  --platform=linux/arm64 .

At build time, the Dockerfile copies a Collector config.yaml into the image, and passes it in as runtime parameter. It’s bare bones and doesn’t test all of the components (in the real world, you probably should feed it a config with all of the components you’re including). I just wanted to make sure that it didn’t blow up completely.

Here’s my bare bones config.yaml:

# Minimalistic OTel Collector Config file to test the Collector build
receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}


processors:
  cumulativetodelta: {}
  batch: {}

  # Prevent out of memory (OOM) situations on the Collector
  memory_limiter:
    check_interval: 1s
    limit_percentage: 70
    spike_limit_percentage: 30


exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ memory_limiter, batch ]
      exporters: [ debug ]
    metrics:
      receivers: [ otlp ]
      processors: [ memory_limiter, cumulativetodelta, batch ]
      exporters: [ debug ]
    logs:
      receivers: [ otlp ]
      processors: [ memory_limiter, batch ]
      exporters: [ debug ]

And after I built my Dockerfile, I fired up my brand-new Collector:

docker run -it --rm -p 4317:4317 -p 4318:4318 \
  ghcr.io/avillela/otelcol-kepler-benchmark-0.102.1:0.1.1 \
  "bin/bash"

Because the config.yaml was copied into the image at build time, I don’t need to specify a config.yaml. But if you can totally override it. So if you want to test it out with a your own config.yaml, you’d run it like this:

docker run -it --rm -p 4317:4317 -p 4318:4318 \
    -v /path/to/config.yaml:/etc/otelcol-contrib/config.yaml \
    --name otelcol \
    ghcr.io/avillela/otelcol-kepler-benchmark-0.102.1:0.1.1  \
    "--config=/etc/otelcol-contrib/config.yaml"

Keep in mind that your Collector image name would be different from mine.

Final Thoughts

I’ve got to admit that this little exercise took a lot longer than I expected. I’m doing this work as part of a talk that I’m giving at KubeCon EU in April, alongside the very awesome Nancy Chauhan, and while this was definitely a deeper rabbit hole than I planned to jump into, I am super grateful for the learning experience. My brain is fried, but my heart is happy, because I learned cool things, and I get to share them with y’all.

And that’s a wrap. Thanks for hanging out. I will now leave you with a photo of lovely, yet rarely-photographed Buffy:

Buffy is enjoying some time with her favourite humans.

Until next time, peace, love, and code. ✌️❤️👩‍💻