Remember to Async

Twice in the last two weeks I’ve run into problems that looked like service issues but can be solved by a simple rule. Remember to async.

The most recent example came with a team that was complaining about slow access to storage queues. When they first mentioned the problem I had waxed poetically about Azure Storage Queues vs Azure ServiceBus vs Azure EventHub. When I got onsite and got into detail it didn’t appear to be a service limitation but something else in the system. They were seeing ~60 puts/second against an Azure Storage Queue within the same region. Storage Queues are rated to about 2000 requests/second so it didn’t look like a service issue at all.

When I asked for more detail they mentioned that they were also seeing dramatic slowdown when they called the queue across regions(as slow as 6 requests per second in one case). After digging a little deeper I discovered that the code they were running looked roughly like this:

foreach (string record in records) {
queue.AddMessage(new CloudQueueMessage(record));

That code looks perfectly acceptable and it’s fully supported in the SDK it’s just never going to run fast.

By making a call to an external service synchronously it leaves itself open to anything transient in the system slowing you down. If they’ looked deeply into response time they’d likely see that one or two requests were slow but most of the requests were returning quite quickly. The problem with synchronous is that they had to wait every time there as 1 slow request.

This is an easy thing to miss. I was chasing a very similar performance challenge all last week before it occurred to me to start thinking through how async could make my life easier.

The fix was to rewrite their code into something similar to what’s below:

Task[] tasks = new Task[records.Count]();
int i = 0;
foreach (string record in records) {
tasks[i] = queue.AddMessageAsync( new CloudQueueMessage(record));
await Tasks.WhenAll(tasks);

Azure Storage Queue’s exposed an Async method so it was easy to just start using this. The loop here will just keep jamming messages onto the queue and only wait at the end for all requests to complete. As a result 1 slow request doesn’t impact the overall speed of the application.

The net result was an improvement to ~350 messages/second. I thought we could probably tune a little more, but it was a pretty good start.

The lesson here’s pretty simple if you’re struggling with performance issues in your code due to external services before you go looking for service issues Remember to Async.

Quick Note:

I don’t expect either of those code samples to work. I wrote them as examples of problem solutions not as copy and paste solutions. Also make sure you put a limiter on the second example before using it. Otherwise you could accidentally spawn a bizarrely high number of threads.