The Case of The Unexplained “TaskCanceledException” and Docker Broken Promise “works on my machine”?
Docker should eliminate the “works on my machine” problem once and for all, right?
First of all, I am a big fan of Docker. Let me repeat.
I am a big fan of Docker.
Well, perhaps I should say, I am a big fan of the containers technology, to be precise. A little bit exaggerated, but I believe containers are the future. And I am not alone.
Docker packages applications and their dependencies together into an isolated container making them portable to any infrastructure. Eliminate the “works on my machine” problem once and for all.
So it’s like “build once run anywhere.” No more worries about missing dependencies during deployments. It should eliminate our concerns about compatibility on different various platforms.
My Use Case
Now let’s dive into my use case.
I had ASP.NET Core 1.1.4 application deployed to the cloud running Ubuntu 17.x. The application was using
HttpClient to call other web services. Then I observed something weird in the log.
Unhandled Exception: System.Threading.Tasks.TaskCanceledException: A task was canceled.
I was able to repro it on every 10 minutes idle of the last
HttpClient call. Then I figured out the minimal time required to repro was 4 minutes. I suspected that it had something to do with TCP/IP TIME_WAIT 240 seconds. Later, I upgraded the application to ASP.NET Core 2.0.3 with the hope that the problem would go away, but with no luck.
I know what is flashing into your mind right now. You would probably suggest me to read this post:
I've been using HttpClient wrong for years and it finally came back to bite me. My site was unstable and my clients…aspnetmonsters.com
Well, no. It was not that infamous
HttpClient usage problem. I have read a bunch of posts about
HttpClient, including it’s potential DNS changes issue:
As described in the following post: http://byterot.blogspot.co.uk/2016/07/singleton-httpclient-dns.html, once you start…github.com
Oh wait, I’m using Docker! I should be able to repro it at my local machine, right?
So, back to the Docker promise: “works on my machine.”
Unfortunately, in this case I was not able to repro it at my local machine running the same Docker engine version 17.x on top of Windows 10. I even tried to set up a new vanilla Ubuntu VM. But still, I couldn’t repro it as well. I also confirmed that both cloud Ubuntu and my VM were using the same
cURL version with
Then I found the same issue reported on GitHub.
This is the raw stack trace from the exception that occurs: FATAL: Unhandled exception (A task was canceled.). DEBUG…github.com
No perfect solution. At the time of this writing. But I don’t care anymore.
Why? ASP.NET Core 2.1 is out. I performed the upgrade and switched to the new HttpClientFactory and its awesomeness has blown all the problems away, including those DNS changes problem. The problem might have related to the native handler
libcurl. The new default from 2.1 is using
SocketsHttpHandler to make behaviors on all platforms uniform.
You can always revert to the old behavior by setting the environment variable:
But again, there’s no looking back.
Pssst! ASP.NET Core 2.1.1 is ultra fast!
(Some of you might notice that this post writing style is somewhat similar to Mark Russinovich’s old blog posts style. 😁 Yes, I am a fan of his!)
About Reacting to DNS changes:
Pooling of handlers is desirable as each handler typically manages its own underlying HTTP connections. This value defaults to 2 minutes:
As written in the comment section, the value should be chosen with an understanding of the application’s requirement to respond to changes in the network environment.
So that’s the weird behavior I was facing during the application development using Docker. Other than that, I am happy with the Docker. In fact, I can’t think of building a new application — especially .NET Core application — without using Docker now. Of course, depends on the requirements, we might have to put Serverless into consideration. Or perhaps, the best of both worlds with something like Serverless Microservices — Service Fabric Mesh?
If you haven’t fallen in love with Docker, you should.
About Me John Willis joined docker from the recent Socketplane acquisition. John is one of the founding members of the…blog.docker.com
Further reading: You might want to check my Docker-related post about LCOW:
HttpClient in ASP.NET Core application prior to version 2.1 might give you different behaviors, even with the use of Docker containers. Upgrading to the latest ASP.NET Core 2.1 makes behaviors on all platforms uniform with the new default
SockectsHttpHandler. Revert to old behavior is possible by setting environment variable:
You should upgrade to the ultra fast ASP.NET Core 2.1, too!
I only write about programming/technology in English and Japanese. If you follow me on Twitter I would try my best to not waste your time.😉