Reaching the Limitations of Linux with Environment Variables

4 min readJan 6, 2024

Before we start, I highly recommend to the reader to have prior knowledge of kubernetes & linux and also a little knowledge of hyperthreading so we could establish common grounds.

I would also like to note that this story slightly revolves around elasticsearch, but it has nothing to do with the limitations.

Background

Shir & Oren are Infrastructure Database Engineers who serve Elasticsearch AAS (ECK) to their customers on massive scales via their kubernetes clusters.

Recently, they discovered that for some reason, basic Linux operations such as pipelines, forks, and subshells take a substantial amount of time to execute.

After a bit more research, it appeared that the readiness probe script which every elasticsearch pod has, and which also runs with a default run period & timeout of 5 seconds, caused high cpu spikes in usage.

For those who don’t know the readiness probe functions as a health check for a pod. This script comes built-in with the ECK operator.

The Journey

Shir & Oren began with debugging the script but sadly had no real leads at first as to why it was happening.

Hours and Days went by, but they just couldn’t understand what the root cause was.

First, they thought that there is a clear correlation between the compute node resource usage and the long execution times, but for some reason, even when nodes were on very high cpu usage, some pods just didn’t experience this issue. It did seem to help cause the issue, but it wasn’t the root cause.

Then they thought that because they had hyperthreading enabled, perhaps that was the reason for the slow cpu operations. On the contrary, elasticsearch is known for being a good contestant for hyperthreading since it does many calls to the RAM and so the second register in the cpu can use the cpu in the meantime. Which means this option was cast out as well.

In addition, as you know, if a pod fails the readiness probe check or times out, kubernetes doesn’t route requests to it. As a result, they had more than one cluster that were inaccessible and so time was against them!

After a while, they decided to consult with a system expert to help them understand what was happening.

The expert used the nsenter command from one of the busier compute nodes in order to try and recreate the issue they were having.

nsenter is a command that allows running commands from a container’s scope (docker is based on it).

After executing the command on the host machine but with the pod’s namespace, to everyone’s surprise, the script ran in milliseconds!

The Solution

So why was the script running so slowly on the containers scope but not on the host’s scope?

Well it turns out that one thing that doesn’t carry over when executing commands with nsenter is environment variables!

When the two created a new kubernetes namespace and deployed a new cluster, suddenly the script ran in milliseconds even on the busiest of compute nodes!

It appeared that having all of the eck clusters in one namespace caused every pod to have around 8k+ environment variables! You might then ask why these numbers accumulated? Well, in kubernetes there’s a service discovery feature that allows containers to discover within their scope other services and communicate with other pods without having root permissions.

After a few minutes of searching later, they discovered the option

‘enableServiceLinks’ was enabled by default.

The two then hastily proceeded to contact the elasticsearch support team, which, after confirming with the dev team, said it was OK to disable this setting.

Disabling this option on crucial tier 1 clusters indeed fixed the problem, and so the two have successfully averted the crisis!

Into the Kernel and Beyond!

After getting a breather, the two didn’t stop there and decided to dive deeper and use the strace command. Using this command showed that there was a specific syscall that took longer than the others to execute. This syscall was execve. After diving a bit into Linux’s Kernel code, we noticed that there was a specific call within execve that took the most.

This call was made to copy_strings, which had two nested while loops, thus explaining why having more environment variables caused our graphs to increase with a slope close to x^2.

BashING our heads

After a long while,

The two discovered that this happened only on a very specific release of bash!

Unfortunately, this specific bash release was used throughout the releases of the elasticsearch docker image (more details on that later)!

The End?

Finally, this concluded their search for the real culprit and thus has ended their long journey!

We have learned a lot during this and hope that you might have caught some in the process too!

Summary

Due to systematic limits, having a lot of environment variables (thousands to tens of thousands) may slow down substantially your basic cpu operations (depending on your hardware) when using a specific version of bash.

In addition, you shouldn’t have all your workloads in one kubernetes namespace (not pointing fingers) when deploying applications in massive scales.

We encourage you to recreate this on your vm / container and see for yourself!

Note that recreating this issue on stronger cpu like the m2 proved to be much more difficult due to it being very fast.

Our tests were done on the following cpu:

Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz

The elasticsearch docker images compared to the bash release, which had these consistent impacts (ltr):

7.17.9 - 5.0.17(1)-release

8.4.1 - 5.0.17(1)-release

8.8 - 5.0.17(1)-release

8.10.3 - 5.0.17(1)-release

8.11.1 - 5.0.17(1)-release

Reaching the Limitations of Linux with Environment Variables

Background

The Journey

The Solution

Into the Kernel and Beyond!

BashING our heads

The End?

Summary

Links

enableServiceLinks:

Linux Kernel Code:

Hyperthreading:

Written by Oren Kessler

Responses (1)