The Case of The Unexplained “TaskCanceledException” and Docker Broken Promise “works on my machine”?

Docker should eliminate the “works on my machine” problem once and for all, right?

Unhandled Exception: System.Threading.Tasks.TaskCanceledException: A task was canceled.

First of all, I am a big fan of Docker. Let me repeat.

I am a big fan of Docker.

Well, perhaps I should say, I am a big fan of the containers technology, to be precise. A little bit exaggerated, but I believe containers are the future. And I am not alone.

Why Docker?

From: https://www.docker.com/what-docker:

Docker packages applications and their dependencies together into an isolated container making them portable to any infrastructure. Eliminate the “works on my machine” problem once and for all.

So it’s like “build once run anywhere.” No more worries about missing dependencies during deployments. It should eliminate our concerns about compatibility on different various platforms.

My Use Case

Now let’s dive into my use case.

I had ASP.NET Core 1.1.4 application deployed to the cloud running Ubuntu 17.x. The application was using HttpClient to call other web services. Then I observed something weird in the log.

Unhandled Exception: System.Threading.Tasks.TaskCanceledException: A task was canceled.

I was able to repro it on every 10 minutes idle of the last HttpClient call. Then I figured out the minimal time required to repro was 4 minutes. I suspected that it had something to do with TCP/IP TIME_WAIT 240 seconds. Later, I upgraded the application to ASP.NET Core 2.0.3 with the hope that the problem would go away, but with no luck.

I know what is flashing into your mind right now. You would probably suggest me to read this post:

Well, no. It was not that infamous HttpClient usage problem. I have read a bunch of posts about HttpClient, including it’s potential DNS changes issue:

Then…

Oh wait, I’m using Docker! I should be able to repro it at my local machine, right?

So, back to the Docker promise: “works on my machine.”

Unfortunately, in this case I was not able to repro it at my local machine running the same Docker engine version 17.x on top of Windows 10. I even tried to set up a new vanilla Ubuntu VM. But still, I couldn’t repro it as well. I also confirmed that both cloud Ubuntu and my VM were using the same cURL version with curl --version.

Image courtesy of https://blogs.gartner.com/richard-watson/ok-get-dockers-great/

Then I found the same issue reported on GitHub.

No perfect solution. At the time of this writing. But I don’t care anymore. 
Why? ASP.NET Core 2.1 is out. I performed the upgrade and switched to the new HttpClientFactory and its awesomeness has blown all the problems away, including those DNS changes problem. The problem might have related to the native handler libcurl. The new default from 2.1 is using SocketsHttpHandler to make behaviors on all platforms uniform.

You can always revert to the old behavior by setting the environment variable:

DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0

Source code:

https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.cs#L17-L37

But again, there’s no looking back.

Case closed!

Pssst! ASP.NET Core 2.1.1 is ultra fast!

(Some of you might notice that this post writing style is somewhat similar to Mark Russinovich’s old blog posts style. 😁 Yes, I am a fan of his!)


About Reacting to DNS changes:

Pooling of handlers is desirable as each handler typically manages its own underlying HTTP connections. This value defaults to 2 minutes:

https://github.com/aspnet/HttpClientFactory/blame/master/src/Microsoft.Extensions.Http/HttpClientFactoryOptions.cs#L19

As written in the comment section, the value should be chosen with an understanding of the application’s requirement to respond to changes in the network environment.



So that’s the weird behavior I was facing during the application development using Docker. Other than that, I am happy with the Docker. In fact, I can’t think of building a new application — especially .NET Core application — without using Docker now. Of course, depends on the requirements, we might have to put Serverless into consideration. Or perhaps, the best of both worlds with something like Serverless Microservices — Service Fabric Mesh?



Summary

Use of HttpClient in ASP.NET Core application prior to version 2.1 might give you different behaviors, even with the use of Docker containers. Upgrading to the latest ASP.NET Core 2.1 makes behaviors on all platforms uniform with the new default SockectsHttpHandler. Revert to old behavior is possible by setting environment variable:

DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0

You should upgrade to the ultra fast ASP.NET Core 2.1, too!


I only write about programming/technology in English and Japanese. If you follow me on Twitter I would try my best to not waste your time.😉

Like what you read? Give Joni a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.