Resilient Http Calls using Polly.

This is a four-part series blog post on using Polly for resilience that you may require for .NET APIs or Services. In this first blog post I detail out what Polly is and its features and demonstrate a basic retry mechanism in .NET. In the subsequent posts I shall discuss some advanced mechanisms that are available in Polly which you may want to take advantage of.

What problems are we trying to solve?

In this day and age most of us develop/write applications in the form of microservices. Without bickering into whether ones require microservices or not, the inevitable truth is that we make lots of inter service calls i.e., calls among services within the application e.g., catalog service, product service, order service etc. We could also have calls made to external services e.g., call to an online payment platform from the payment service.

Given the nature of the network that we operate on and the types of bugs that we introduce while developing applications, the chances are that sometimes the service calls are not fulfilled i.e., they fail. So, what do we do, we have some of the following options.

• Retry and retry after some time.
• Have a fallback or backup mechanism if the retries fail.
• Return results from cache.
• Brask circuit and prevent further calls.

and many more.

We could develop the aforementioned functionalities by ourselves or use a battle tested resilience framework. E.g., you could use a popular service mesh framework like Istio or Linkerd or may be even something like Dapr.

Polly is one such reliance framework which helps integrate resilience to .NET applications.

What is Polly?

Read the details here!

What are the Polly strategies?

Polly has two main strategies.

• Reactive
• Proactive

Reactive Strategy includes the following.

1. Retry
2. Wait and Retry
3. Circuit Breaker
4. Fallback

Proactive Strategy includes the following.

1. Timeout
2. Caching
3. Bulkhead isolation.

Mostly these Polly policies are used in combination although it's possible to use one of them.

Using the HttpClientFactory instead of the HttpClient

All the examples here are based in .NET 5+. The ASP.NET team introduced the HttpClientFactory to be used as a best practice as opposed to HttpClient for very good reasons and you too should be using the same.

Refer the following article on why you should be using HttpClientFactory and the different types which you may want to consider using.

Use the IHttpClientFactory — .NET | Microsoft Learn

Here’s how I’ve integrated HttpClientFactory in my codebase.

services.AddHttpClient<IInventoryClient, InventoryClient>().ConfigureHttpClient(
    (serviceProvider, httpClient) =>
    {
        var httpClientOptions = serviceProvider.GetRequiredService<InventoryClientOptions>();

        httpClient.BaseAddress = httpClientOptions.BaseAddress;
        httpClient.Timeout = httpClientOptions.Timeout;
    });

I’ve also used a mechanism on how to use IOptions patterns with validation on startup. Probably that's another blog for a later time.

Basic Polly integration with retries policy.

The application contains two Web APIs namely order service and inventory service. Both these are developed using ASP.NET Core 7. Each of them has a single controller class and a method each.

To trigger an error or to return a 500 status code I’ve modified the inventory service to fail 4 out of 5 times. (Meaning when multiple requests are sent only every 5th request will succeed. I’ve also got a static counter in the InventoryController to keep track of the number of requests sent. I’ve also introduced a delay of 1 second to mock a typical response from an API.

You need to add the following NuGet to the project concerned, which at the time of the writing is at 7.0.10.

<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="7.0.10"/>

In the order service Program.cs file change/update the registration of inventory client as shown in the code.

// Create the retry policy we want
var retryPolicy = HttpPolicyExtensions.HandleTransientHttpError() // HttpRequestException, 5XX and 408  
    .RetryAsync();

// Register the InventoryClient with Polly policies
services.AddHttpClient<IInventoryClient, InventoryClient>().ConfigureHttpClient(
    (serviceProvider, httpClient) =>
    {
        var httpClientOptions = serviceProvider.GetRequiredService<InventoryClientOptions>();

        httpClient.BaseAddress = httpClientOptions.BaseAddress;
        httpClient.Timeout = httpClientOptions.Timeout;
        
    }).AddPolicyHandler(retryPolicy);

In the above code, we initially define the retry policy once in the event of a transient error. Thereafter register the policies with the HttpClientFactory.

Transient errors are those that belong to the 400 series and 500 series errors.

To mock the failure of a calling service, you’ll need to modify the Inventory Controller class with the following code. This fails the first request and succeeds in the second request with 200 status code.

private static int _requestCount;

[HttpGet("{productId}")]
public async Task<IActionResult> Get(string productId)
{
    await Task.Delay(1000); // simulate some data processing by delaying for 100 milliseconds 
    _requestCount++;

    var product = _products.FirstOrDefault(p => p.Id == productId);

    return _requestCount % 2 == 0
        ? // only one of out four requests will succeed
        Ok(product)
        : StatusCode((int)HttpStatusCode.InternalServerError, "Something went wrong");

 
}

Start both the Web APIs and send a GET request to the order service, which in turn calls the inventory service requesting for a product detail by Id. Keep the program console open to see the failure in the first call and another request then succeed in the next call. This magic is performed by Polly.

You can also try multiple retries with the following modification to the Polly policy.

// Create the retry policy we want
var retryPolicy = HttpPolicyExtensions.HandleTransientHttpError() // HttpRequestException, 5XX and 408  
    .RetryAsync(4);

Running both Web APIs again, you’ll see the following output,

Polly supports fixed timed retries and exponentially timed wait and retries Fixed timed retries are the ones that I have demonstrated above, however you could also make Polly wait and retry and a given interval e.g., after a minute or so. The following code depicts how a retry could be triggered every 10 seconds for any number of time (in this case 4 times).

// Create the retry policy we want
var retryPolicy = HttpPolicyExtensions.HandleTransientHttpError() // HttpRequestException, 5XX and 408  
    .WaitAndRetryAsync(4, retryAttempts => TimeSpan.FromSeconds(10));

Restarting both the services you should see something like the following with each request having a gap of 10 seconds between them. Take a look at the time the requests are made to the inventory service. They have an interval of 10 seconds between them.

Next, we also could add some exponential delay or jitter between the API calls. This means the time interval between calls increases exponentially.

var retryPolicy = HttpPolicyExtensions.HandleTransientHttpError() // HttpRequestException, 5XX and 408  
    .WaitAndRetryAsync(4, retryAttempts => TimeSpan.FromSeconds(Math.Pow(2, retryAttempts)/2));

Here’s how the code should look like and here’s the log in the console. Take a look at the time intervals.

References

This has plenty of examples and documentation and demonstrations, most of the content in this and the subsequent blogs are based on this video tutorial on Pluralsight by Bryan Hogan

You could also get some insights into Polly by going through the official GitHub page.

Code

The code related to this blog could be found here. The main branch contains the final code which encompasses illustrations and descriptions of all the four blogs. If you are only interested in the code related to this particular blog, then switch over to wait-and-retry-mutiple-times branch.