Contexts, Goroutines, and Memory Leaks

John Cai
John Cai
Aug 25, 2017 · 2 min read

At Seed we have a service that sends scheduled requests to configured URLs. To implement clean shutdown, we decided to take advantage of Go’s context package.

That way, when we get a shutdown signal we can tell the workers to cancel the requests they are sending and be able to return immediately.

In this code, we have a goroutine that fires off an http request with a context. Then we listen on the doneChanfor the result. Or, if we get a signal from the context from a cancellation or a timeout, we exit early. In this way, we don’t need to wait for the request to come back when we initiate a shutdown.

The Problem

This is a memory profile of our service. There’s a serious memory leak happening here. When it gets to 100% in this graph, ECS is killing the container and recreating it.

The Leak

Here is what was happening

Notice that when we get a context cancellation, we will exit at line 28. However, this leaves doneChanhanging with nothing listening on it. This means that the send on doneChanin line 15 will block forever. This means the defer res.Body.Close() never gets run, which means these readers never get garbage collected.

The Solution

The change is as simple as using a buffered channel instead of an unbuffered channel. With a buffered channel the send in line 15 will not block even if nothing is listening to doneChan.

)
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade