Tracking Down an ETS-related Memory Leak

Tyler Pachal
Sep 27 · 7 min read
A callback which eventually gets call by Brod to handle Kafka messages. Looks simple, but there is a memory leak here!
DataDog: memory consumption is very high
Datadog: Most memory being using by Binaries, not ETS
Example of :erlang.memory/0 in action
# Inside the machine# Find the CONTAINER ID of your Elixir service
$ docker ps
# Enter inside the container
$ docker exec -it <CONTAINER_ID> bin/bash
# Start a remote_console, you maybe have to set other args and change the path to the binary
$ /opt/app/bin/my_service remote_console
Using :observer_cli while attached to a service in production to get memory information about each process
Using :recon.bin_leak/1 to see the top memory offenders

Jason source code
An example showing that inserting into ETS will increase the reference counter of a binary. Annotated code version here.
An example of `:binary.copy` and the resulting binaries on the process

Datadog: The binary memory (dark blue) has stabilized, but now my ETS memory (light blue) is growing unbounded 😛

Tyler Pachal

Written by

Software engineer at PagerDuty, working with Scala and Elixir.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade