That one time without the mutex

Peter Naulls
Nov 30, 2019 · 3 min read

Every developer who starts with threading soon learns about the necessity of locking structures. The reason is pretty simple — if your program has two threads of execution, and there’s any chance at all one of them will alter the structure, then you need to lock between them.

Image for post
Image for post

The exact way you do this varies a bit by language, but the old school way of doing this in C and C++ is with a boring mutex, which you claim before you access the structure and then release after.

Without such locking, there’s a chance that one of threads can alter the structure whilst the other is part way through reading — if that’s pointer, or there’s assumptions about the atomicity of the values, well, bad news if you can’t control the sequencing. Even if it’s “incredibly rare”, or “one in a million” or whatever you want to tell yourself, it’s probably going to happen, and the consequences and likely crash are going to be unpredictable.

You really need that lock

As an aside, it is sometimes (very occasionally) possible to get away without full locking and instead make use of simple flags. Such cases have to be engineered very carefully, and are most suited to situations where you’re OK with sometimes having junk in a buffer (e.g, video transitions). Even so, it can be hard to guarantee things.

The mutex works, because it ensures that only thread can claim the mutex at once — this is done (at least in old school computer architecture terms) by disabling interrupts and therefore thread context switches during the claim.

Novice software developers starting with threads learn at least some version of the above, or least they understand that data can be shared by threads and that’s “bad”.

So, what happened?

(Some details obfuscated to (a) avoid strict identification and (b) because I forget some of the precise details and (c) because I feel like it)

I was running some recent code from a co-worker through a memory checker (valgrind) and spotted some corruption. Upon code inspection, it was obvious it was one these cases; and upon further check, it was obvious that a similar section of code using a different structure was also at risk. And so, I did what and good developer would do, and politely pointed these out. As some background, this was a mid-level developer known for over-complex solutions.

What was his response? Reluctant agreement with the first section of code, but firm denial of the second section. His reasoning? “QA didn’t report a problem with it”.

*facepalm*

Now, I can already hear many of you already lining up your objections, but let’s lay this out fully:

  1. QA indeed hadn’t seen it — their focus had been on an older version of the code for a release candidate and would be for a while longer.

When I went on to explain that lock requirements are necessary, he wanted everyone to gather around a large screen and do a code review. It’s unclear what he hoped to achieve with this.

When my patience eventually ran down, the baton was taken up by a co-worker. He pointed out rightly that there might be some performance concerns here — indeed, locks can be “expensive” in some cases — but the developer didn’t take up this line of argument, and in practice, this wasn’t empirically supported.

The argument was eventually “won”, but probably not for the right reasons.

Conclusions

This is of course a classic case of the cost of fixing bugs. In the classic book, “The Mythical Man Month”, it’s pointed out that the time cost of fixing bugs at each stage of software lifecycle goes up exponentially:

  • Fixing when I had pointed it out (or the original developer doing the right thing) — minutes.

And so it goes. Always check your locks.

Adventures in Software Development

Programming, Software and Lessons Learned.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store