Unmasking the Memory Menace

Memory leaks in Go (Golang) occur when a program allocates memory but fails to release it when it is no longer needed. Go has a garbage collector that automatically manages memory for most cases, but memory leaks can still occur due to certain programming practices.

As we build our system under emerging design principles, such considerations are not believed to be of importance and that is okay. It is more important to build the system in a way that avoids premature optimizations and enables you to perform them later on as the code matures, rather than over-engineering it from the get-go. Still, some common examples of seeing memory pressure issues materialize are:

• Too many allocations, incorrect data representation.
• Heavy usage of reflection or strings.
• Using globals.
• Orphaned, never-ending goroutines.
• Unordered declaration of structs.

in redBus I have noticed that the memory is increasing consistently with the increase in time which definitely suggest that our application might have some unknown leak.

Memory Utilization with Memory leak:

Memory Utilization

No of requests served per day

I started doing some analysis in terms of code review , api error , timeouts,no of connection, response size and so many other things which can cause memory leak but no luck still seeing the same issue on memory.

Harnessing the Power of Go’s pprof: Unveiling Memory Woes Through Profiling and Analysis

Using pprof to identify the root cause of memory issues in your Go application is a powerful technique. Pprof, short for “Performance Profiler,” provides a way to gather and analyze runtime performance data, including memory usage. In this context, we’ll focus on how pprof helps in identifying memory-related problems and how it does so using profiles.

Heap Profile:

• The heap profile in pprof is instrumental in understanding memory allocation and usage. It provides insights into both current and cumulative memory allocations.
• Pprof generates a heap sampled dump file, which is essentially a snapshot of your application’s heap memory at a particular point in time.
• This snapshot allows you to see which parts of your code are responsible for memory allocations.
• By analyzing the heap profile, you can pinpoint memory leaks or excessive memory consumption.

Goroutine Profile:

• The goroutine profile helps you understand the state of your goroutines (concurrent execution units in Go) and the stack traces associated with them.
• This profile includes stack traces of all current goroutines, making it valuable for diagnosing issues related to concurrent execution, like deadlock or excessive goroutine creation.

Profiles and Events:

• Pprof operates by collecting profiles, which are essentially collections of stack traces. These stack traces show the call sequences that led to instances of specific events, such as memory allocations or goroutine creation.
• For memory-related issues, you primarily use the heap and goroutine profiles.
• The pprof package in Go, particularly the runtime/pprof package, facilitates the creation and management of these profiles.

Built-in Profiles:

• Go offers several built-in profiles that cover common performance monitoring scenarios:
• goroutine: Provides stack traces of all current goroutines.
• heap: Gives you a sampling of memory allocations for live objects.
• allocs: Offers a sampling of all past memory allocations.
• threadcreate: Shows stack traces leading to the creation of new OS threads.
• block: Provides stack traces leading to blocking on synchronization primitives.
• mutex: Contains stack traces of goroutines holding contended mutexes.

Using Heap and Goroutine Profiles for Memory Issues:

• When dealing with memory issues, you’ll primarily focus on the heap and goroutine profiles.
• The heap profile will help you identify which parts of your code are responsible for allocating memory and whether memory is being freed as expected.
• The goroutine profile can be crucial in identifying goroutine leaks or understanding why certain goroutines are blocked, which might lead to memory consumption issues.

Setting up pprof:

We can set up pprof by two ways. There are two main ways of obtaining the data for this tool. The first will usually be part of a test or a branch and includes importing runtime/pprof and then calling pprof.WriteHeapProfile(some_file) to write the heap information and the second method is If your application is a web app that already has an HTTP server running, you can just import the pprof package and If you don’t have an HTTP server running, you need to start it too, so you can add this to your code

Since in our code base we were not using the DefaultServeMux we need to register the route for pprof like this:

muxServer.HandleFunc(“/debug/pprof/”,pprof.Index)

Now we can extract the dumps for our application.

Extracting a dump

Now you have different endpoints that will give you different profiles. To see the available profiles you can make a GET to the endpoint.

For Example: http://localhost:8081/debug/pprof

Finally after setting up pprof I analyse some memory parameters and plotted the memory values using a bar chart.We can clearly see our heap inuse and heap release is nearly same which means our GC was running fine but our heap allocs are increasing with time which definitely suggest that our memory is increasing consistently.

All values are in bytes

Then later I analysed goroutines profile there I saw we have goroutines whose number were keep on increasing with the time and we have orphaned go routines which were not properly terminated.Basically Orphaned goroutines are goroutines that are no longer reachable from the main program or other active goroutines but have not been explicitly terminated and can indeed continue to reside in memory. Specifically, they are not collected by the Go runtime’s garbage collector until they have exited.

As we can clearly see in the above image in the file wallet.go we have a maximum number of goroutines which are not being terminated properly and due to accumulation of these orphaned goroutines in the heap they are increasing memory.

After fixing the goroutine and removing some global variables which were defined we were able to fix our memory issue.

Our memory graph looked something like this.

Final Notes

In conclusion, I have shared the experience that is currently at the forefront of my mind. This experience contributed to successfully deploying the system and fulfilling most of the use case requirements. Thank you for your attention.

References

Golang Doc.

Grafana for metrics https://grafana.com/