Why goroutines leaks exist, and how to avoid these

In this age, software programs and applications are expected to behave quickly and smoothly. Being able to run code parallel is one of today’s capabilities to meet this need. But beware of the danger of leaks! We talk about it with Jacob Walker from Ardan Labs, who dedicated a blog to it for Golang engineers.

In advance of starting to read our chat about debugging Goroutines leaks, first a small introduction of a couple of principals that give a which will give you a more general picture of the challenge this concept solves.

Concurrent programming:

Concurrent programming is a parallel coding approach that enables a collection of consecutive threads to be run simultaneously. In this way, a software program can compute faster and as such is able to perform better. Concurrent programming is empowered by the capabilities of today’s multicore processors.

Go threads meeting concurrency

Goroutines:

Conventional threading approaches are based on communication between threads that use shared memory. Instead of specifically make use of locks to mediate access to shared data, Go stimulates the use of channels to move references to data between goroutines. In this way, only one goroutine has access to the data at a given time. Golang meets the needs of concurrent programming with these goroutines, which are basically light weight threads controlled by Go’s runtime.

Goroutine channels pipes concurrency

Leaks

But beware! Goroutines can slowly but surely stockpile over a certain period and as such waste your memory resources, without you even noticing it. Therefore it’s very important you are aware of the danger and/or existence of leaks and debug them early. This the topic we discussed recently during an interview with Jacob Walker, within the #live-interviews channel of the Gophers Community at Slack (40.000+ members). Read the interview here below.

interview jacob walker ardan labs golang goroutine leaks
To summarize I have been working as a developer for about 10 years and have been focused almost exclusively on Go for the last 4 years. I started around version 1.3.
I work at Ardan Labs as a Community Engineer. My primary activity is teaching classes like Go Fundamentals and Ultimate Go. In between classes I work on blog posts, developing new content, and helping people from the community.

Sebastiaan (interviewer) [4:16 PM]:

What are the reasons why you have embraced Go the past 4 years? What are the main reasons that you assumingly love the language?
Jacob Walker [4:17 PM]:
Ha, yes I definitely love it :slightly_smiling_face: My formal education is in business (MBA). So when I started development I was mostly "self-taught" using resources online. I started my career path doing web applications with HTML, JS, then PHP. That served me really well for several years and I was able to make some cool stuff and solve some interesting problems. After about 5 years of that and dabbling with other languages like Ruby and Python I wanted to go deeper and work closer to the machine. I looked at C and C++ and experimented there for a while but Go just clicked for me. Something about the mindset and philosophy of Go just really matched up with what I was looking for and how I wanted to write code. As for reasons I love the language? Man there are quite a few :slightly_smiling_face: - Simplicity of both the language and the code and patterns it encourages. - Consistency. The way the rules are pretty consistently applied across the language. - The tooling is phenomenal. - The general idea that there is usually "one right way" to do most things so I don't have to spend a lot of time second-guessing which of a dozen ways I should approach a problem Those are the first that come to mind.

Sebastiaan (interviewer) [4:25 PM]:

If you look back to this year, and you (and Ardan Labs) being an important contributor/user regarding Go, with what kind of feeling do you look back to the progress that is made with the idiom?
Jacob Walker [4:26 PM]:
We have seen an explosive growth in the user base. Demand for trainings from companies just keeps going up. It has been a very exciting time for the language.

Sebastiaan (interviewer) [4:27 PM].

Ok, and could you elaborate a bit more on this ‘exciting’ time? What makes it specifically exciting?
Jacob Walker [4:30 PM]:
Yes. To me it is exciting to see all of these new gophers joining the community! I enjoy the language and it is always nice to share things you enjoy with others. More and more people are starting to understand what it is about the language and its ecosystem that seasoned gophers have loved for so long.
Everything is just looking up right now.

Sebastiaan (interviewer) [4:32 PM]: W

What kind of new features do you personally hope to see released in 2019?

Jacob Walker [4:33 PM]:
Hmm that's a good question. There has been a lot of talk recently about "Go 2" and the draft designs for "generics", simplified error handling, error context etc. Those kinds of discussions are always interesting.
I haven't worried too much about the specifics of those new features though because I know
1. they can still be a long ways off
2. they may look significantly different when they're released compared to the current designs
In the more immediate future I am probably most excited to see the Modules story get solidified.

Sebastiaan (interviewer) [4:38 PM]:

So you wrote about Goroutine Leaks. Why did you chose this topic? Is it a challenge you a lot of times see not been tackled by junior gophers?

Jacob Walker [4:41 PM]:
Yes, definitely. Earlier when you asked about what I love about the language I didn't list Concurrency for a reason. I definitely do enjoy writing concurrent code and Go has one of the best approaches to concurrency I have seen. The problem is many new gophers use it when it isn't necessary and adding concurrency comes with some traps. In my training class when we get to concurrency I caution my students to be aware of these traps and how to avoid them.
This information is useful for everyone in the community so I thought to transform it into a series of blog posts. The Goroutine Leaks post you linked before is the first in that series.

Sebastiaan (interviewer) [4:44 PM]:

Which traps do you feel are the ones that developers overlook the most?

Jacob Walker [4:45 PM]
Well there are Goroutine Leaks which are very easy to accidentally create. The next most common I would guess is Data Races. (edited)
There is also the risk of Incomplete Work. You can start a Goroutine to do something but it might not be done when your program terminates and then it gets cut off. (edited)
It's hard to say which is the most common or the most dangerous. They're all bad :slightly_smiling_face:

Sebastiaan (interviewer) [4:49 PM]:

Towards these traps, do you have any code recipe you may recommend to work safely with concurrency in Go?

Jacob Walker [4:51 PM]
Sure. You can address Goroutine Leaks and Incomplete Work by remembering the mantra
> Never start a goroutine without knowing how it will stop.
As far as I know this saying comes from Dave Cheney's blog post https://dave.cheney.net/2016/12/22/never-start-a-goroutine-without-knowing-how-it-will-stop.
For Data Races the community has the proverb
> Don't communicate by sharing memory, share memory by communicating.
It is kind of pithy and poetic so it might not make sense at first but once you understand the proverb can be helpful for remembering. The key is to avoid sharing the same variable/memory across Goroutines. Rather every variable should be owned and managed by a single Goroutine (edited)

Sebastiaan (interviewer) [4:56 PM]:

So tackling these potential problems is based on following basic principles. If you work in a team, are there certain best-practices to ensure you always follow these principles?
Jacob Walker [4:58 PM]:
All I can really say is to practice Code Review. Nobody should be releasing code to production that isn't thoroughly reviewed by at least one other person. Make sure everyone on the team knows these principles and knows to look for them.
For the first proverb it will get you pretty far to ask yourself "when will this goroutine terminate?" every time you see the `go` keyword.

Sebastiaan (interviewer) [5:01 PM]:

With regards to security, to what kind of problems can leaks lead?
Jacob Walker [5:03 PM]:
The only thing that comes to mind is creating a vector for a Denial of Service attack. If an attacker knows that making a request in a particular way causes the program to leak a goroutine then they could quickly cause the server to leak so many goroutines that the server runs out of resources and crashes.

Sebastiaan (interviewer) [5:06 PM]:

Are there additional resources with regards to reviewing and security that enable you to deal with goroutine leaks?
Jacob Walker [5:09 PM]:
Sure. Code Review before something goes out is best but once the code is out in production or even in a staging or QA environment you can use monitoring tools like `pprof` to count the number of active Goroutines. If that number always increases and never decreases then you have a leak somewhere.

Sebastiaan (interviewer) [5:10 PM]:

Great advice. Let’s include a link about that: https://golang.org/pkg/net/http/pprof/. Are there additional articles from Ardan labs you would recommend regarding the topic we discuss?
Jacob Walker [5:13 PM]:
Bill recently finished a series about the mechanics of the goroutine scheduler. This isn't about leaks specifically but it provides great material for understanding how the scheduler behaves when you start using concurrency. https://www.ardanlabs.com/blog/2018/12/scheduling-in-go-part3.html (edited)

Sebastiaan (interviewer) [5:15 PM]:

And you already told your article is the start of a serie what can we expect to be discussed too?
Jacob Walker [5:15 PM]:
I have already released a second post showing more examples of Goroutine Leaks https://www.ardanlabs.com/blog/2018/12/goroutine-leaks-the-abandoned-receivers.html

Goroutine Leaks are a common cause of memory leaks in Go programs. In my previous post, I presented an introduction to Goroutine leaks and provided one example of a common mistake that many Go developers make. Continuing that work, this post presents another scenario on how Goroutines could be leaked.
I might continue on with Goroutine Leaks or I might move on to the other traps of concurrency. Data Races, Incomplete Work, and finally Unnecessary Complexity.

Each of those topics will take at least one or possibly multiple posts to discuss. (edited)

Sebastiaan (interviewer) [5:18 PM]:

From all these topics/challenges, which one have you personally encountered to be problematic to deal with, or even to be daunting?
Jacob Walker [5:21 PM]:
I have encountered each of these and they all come with a different kind of difficulty so it is hard to say which is the most problematic. Goroutine Leaks and Data Races are especially difficult because you might not even know they are happening. (edited)

Sebastiaan (interviewer) [5:23 PM]

Alright, thanks for this interview @jcbwlkr. When do you approx. expect your next post about concurrency will be published?
Jacob Walker [5:24 PM]:
Oh, now I'll have a deadline! :slightly_smiling_face:
I am shooting for sometime in the next few weeks but sometimes it can take weeks to review these posts to be sure they are up to our standards. So probably by the end of January?

Thank you for having me, @Sebastiaan!
interview 2 jacob walker ardan labs go goroutine leaks
interview 3 jacob walker ardan labs golang goroutine leaks

Originally published at Jexia.