How to investigate .NET memory leaks in Kubernetes

Investigating memory leaks can be hard when you use Kubernetes. Your application is running on transient pods located in the cloud, which makes it hard to access via the usual route of attaching a profiler. If you are to be able to reproduce the issue locally you can consider yourself lucky, but many of the memory issues only arise under specific conditions and tend to reproduce only under actual load, in production only.

While APM solutions like DataDog provide you with an easy way to profile the speed of your application, profiling the memory could be more tricky. Fortunately, Kubernetes and .NET infrastructure provides you with enough tools to be able to take a memory snapshot on your running app on the fly, without disrupting it and without taking much time. In this article I will guide you through simple steps to achieve that.

In the following steps, I’ll be assuming you are running a Docker container based app on a Linux pod, but steps for Windows will be almost identical.

1. First step would be to connect to a shell terminal of a pod. You can do it via GCP, command line or by using the Lens app. I’ll be using OpenLens, a free alternative to the Lens application. Select your pod and click the ‘Open Shell’ icon. You should see a terminal open.

2. Now that you have a remote terminal session established, you can execute the commands necessary to get a memory snapshot. We will be using dotnet-gcdump utility to do that. First command would be to curl download the utility to your pod (use Shift+Ins to paste the text):

curl -L https://download.visualstudio.microsoft.com/download/pr/20221018.2/DC08BD07E116A29B73555284D00C9E8C42AFBFA37A37E3795D6BC8F6FEEDE15B/dotnet-gcdump -o dotnet-gcdump.exe

3. The download normally only takes a second. After that you should use chmod to be able to execute this program (this step doesn’t apply to Windows pods):

chmod +x dotnet-gcdump.exe

4. Now you need to get a process ID of your application. Execute the command below and find the PID of your .NET process in the output. If your application is running as the main app of your container the PID will be equal to 1, but that’s not necessarily the case:

./dotnet-gcdump.exe ps

5. Now use the PID you’ve retrieved to take a memory snapshot of your app and store it into a file. This requires that you have enough free hard disk space on your pod, of course, but snapshots don’t usually take much space:

./dotnet-gcdump.exe collect -p <your_pid> -v

6. Now that the snapshot is located on your pod, you need to transfer this file to your local machine. Copy the generated dump path and file name and then open up a shell on your own computer now and run a command in the following format:

kubectl cp <your_pod_name>:<generated_path_name> ./<generated_dump_name> -n=<your_namespace_name> -c=<your_container_name> — retries=X

If you are not familiar with kubectl, you start by familiarizing youself with the most important commands using the excellent official Kubectl cheat sheet.

7. Now that the dump is at your computer, you can use Visual Studio to open it and examine the contents of your app memory. This is what it may look like. You can see the memory space occupied by each type of objects and investigate the memory paths:

Some additional hints that may help:

• If gcdump results are not verbose enough for you, you could go even further: install an actual profiler on your pod using the same shell terminal and attach it to your app using commands. However, in our experience, we mostly find gcdump output to be sufficient to locate the source of the problem;
• If you encounter “unexpected EOF” error during “kubectl cp” step, you can try increasing the ‘ — retries’ parameter;
• Sometimes gcdump will throw an error during taking the snapshot. A simple retry may help. If it doesn’t, try doing it at a different pod: choose a pod where memory leak is already evident, but memory is not yet completely full, so gcdump has enough free RAM to operate.

Hope this will help you reduce the issues with your .NET apps and good luck with your profiling!

Curious to join us on the mission to empower lodging businesses worldwide? Check out our career page for open positions!