The Dead Remember Their Past Lives

.NET Threads Can Do Scary Stuff

What’s worse than zombies? How about zombies who remember who they were before they became the undead?! Imagine your former friend turned zombie remembers your name and calls it out to you as he tries to get at those delicious brains of yours. Terrifying right? Well this just happened today in real actual life.

Ok, so maybe threads and zombie aren't exactly the same level of terrifying, but they are damn close if you've ever written or, god forbid, debugged multi-threaded code. Well today I had a bit of a scare at work and was once again reminded of how nightmarish it can be. Let’s start at the beginning.

In the world of ASP.NET MVC an http request comes in to the web server and the application spins off a thread to handle that request. No big deal this is something built-in and controlled by the framework so we normally are abstracted away from it and quite safe. Now, a while back when we started work on this particular application we found that there are certain things that come with a web request that are good to have when our application calls down through the business layer and the data layer. In a previous application we had run into the pain of having to pass those items down function by function through all the required layers and soon all our function signatures started to look a little something like this:

Those two strings params appear EVERYWHERE!

This gets old fast and so we discovered a fun way in .NET to pass things through our application without having to pass them into the methods explicitly, thread local storage or TLS for short. TLS allows you to set data on the executing thread which you can then use anywhere in your application so long as you are still running in the context of that particular thread. GREAT! Problem solved.

Fast forward a few months and we start noticing something in our log files. Some of our messages weren't making sense. Errors were happening from weird user/browser combinations. Errors from our system processes were saying that random clients were performing them. Let me rephrase that last bit so you can consider the implications. Code that shouldn't be accessible by any of our clients… appeared to be running in the context of said clients. [insert your expletive of choice here, and here, and here.]

As it turns out, there is something in the .NET documentation that talks about this little situation.

Important

When the thread pool reuses a thread, it does not clear the data in thread local storage ... Therefore, when a method examines thread local storage …, the values it finds might be left over from an earlier use of the thread pool thread.

[Many more expletives.]

Somehow we missed this little tidbit and our threads were holding on to the data when they died and bringing it back with them when they were resurrected from the thread pool. Thankfully, in our case the only issue was a few logging messages being messed up in a small percentage of the situations where we tried to log things, but this could have been an incredibly painful mistake.

We have since corrected our application to tear down the threads before sending them back to the thread pool, but let this be a warning to anyone out there using TLS. The dead have long memories and if you don’t take them out properly they may just come back to haunt you.