Go’s append is not always thread safe

Jack Lindamood
Jun 12, 2018 · 3 min read

Problem Example

I commonly see bugs appending to slices in Go in a way that is not thread safe. A simple example is the unit tests below. This test has two goroutines append to the same slice. If you run this test with the -race flag, it works just fine.

Now, let’s change the code just slightly to create the x slice with more spare capacity. The only thing that changed was line 9.

If we run this test with the -race flag, we will notice a race condition.

< go test -race .

Explaining why this test fails

To understand why this fails, look at the memory of x in the old example.

Image for post
Image for post
x starts with no capacity to change

Go notices that there is no memory to place "hello", "world" or to place "goodbye", "bob", so it makes new memory for y and z. Data races don’t happen when multiple threads read memory, x, that doesn’t change. There’s no confusion here, so there is no race.

Image for post
Image for post
z and y get their own memory

Things are different in the new code.

Image for post
Image for post
x has capacity for more

Here Go notices that there is memory to place “hello”, “world”. Another goroutine also notices that there is memory for “goodbye”, “bob”. The race happens because both goroutines are trying to write to the same spare memory and it’s not clear who wins.

Image for post
Image for post
Who wins?

It is a feature, not a bug, that append does not force new memory allocations each time it is called. This allows users to append inside a loop without thrashing garbage collection. The downside is that you have to be aware when appends happen to the same original slice from multiple goroutines.

Cognitive root of this bug

I believe this bug exists because Go has, for the sake of simplicity, put many concepts into slices that are usually distributed. The thought process I see in most developers is:

  1. x = append(x, ...) looks like you’re receiving a new slice.
  2. Most functions that return values don’t mutate their inputs.
  3. Often when I use append the result is a new slice.
  4. This leads one to, falsely, think append is read only.

Identifying this bug

Pay special attention if the first variable to append is not a local variable.This bug usually manifest when append is happening to a variable stored inside a struct or a variable passed into the current function. For example, a struct could have default values that are appended to each request. Be careful when appending to shared memory, or memory the current goroutine doesn’t entirely own.

Workarounds

The easiest workaround is to not use shared state as the first variable to append. Instead, make a new slice with the total capacity you need, and use the new slice as the first variable to append. Below is the failing example test modified to work. An alternative to append here is to use copy.

append with the local variable first

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store