Quality check list for .NET microservices on Azure

As a front-end developer, I somehow find myself reviewing a lot of dot NET micro-services source code that is deployed to Microsoft Azure. Micro-services has allowed Prospa to scale its engineering department from ~15 in one Sydney office to over 50 in multiple locations. It has allowed development teams to collaborate smoothly and certainly contributed to the success of the company. However, it has also caused some surprises in production due to inconsistency: in quality amongst service components, monitoring and deployment methods.

This article details some of the opinions we formed over .NET Core repositories from our experiences of developing and managing over 200 ASP.NET Core applications deployed on Azure’s Cloud Platform.

Many of these opinions come from pain points we experienced as well as the quality of life improvements we gained from implementing them consistently.

Keep application configuration out of the ARM template and App Service Configuration

Reason: Every time configuration is changed in an App Service, the application will be restarted and can cause unnecessary downtime. Furthermore, the application becomes less portable and harder to test locally because the configuration is split over multiple places.
The app service configuration should be kept to the minimum of:

• WEBSITE_TIME_ZONE
• DOTNET_ENVIRONMENT
• ASPNETCORE_ENVIRONMENT
• APPLICATIONINSIGHTS_INSTRUMENTATIONKEY

Set the time zone on your app services

Reason: It is best practice to keep all environments as similar as possible — including your local development environment. If you perform any operation with DateTimes, you will reduce the chance of bugs.

Consider the example of an Azure Webjob’s timer trigger.

Unexpected behaviour will occur if the timezone is not set.

// When running locally, will report it runs at 12 pm AEST every day.
// if no time zone is explicitly configured on the app service, it will default to UTC 
// and unintendedly run at 10 or 11 pm AEST.
public async Task PerformDailyJob([TimerTrigger("0 0 12 * * *")]TimerInfo timer)
{
}

All Azure resources must be tagged with some kind of owner

Reason: Tagging allows common monitoring and infrastructure to scale across hundreds of services. When present, these tools can be centrally configured to read these tags and notify the appropriate teams of issues such as TCP connection abuse or exceptions. If there are no tags, any issues that occur to these apps will not be fixed until we receive customer complaints.

Configure build and release definitions as code

Reason: Best practices such as blue green deployment, automated readiness checks and unit testing is much easier to implement consistently when they are configured as plain text file in source control. Copying (or referencing) text files and updating a few app specific variables is far less frustrating and error prone than painstakingly comparing many text boxes and menus. When configured manually in the UI, many of these essential steps were missing or misconfigured.

Further, changes to the build and release definition can more easily be changed since different branches can have different definitions.

Verify the health of your app after deployment

Reason: Deployments to Azure App Services aren’t 100% reliable. Sometimes they fail for a variety of reasons. When readiness checks are incorporated into each release definition it prevents unnecessary downtime from failed deployments and reduces user impact.

Make sure diagnostics and logging are configured correctly

Reason: When apps are instrumented correctly with logging, tracing and metrics; finding and fixing production issues becomes very easy.

Dotnet, has a simple but powerful class for storing diagnostics context and exposing it to logging system called System.Diagnostics.Activity. This diagnostic context can even flow across different processes.

// any application logs and traceswill automatically get tagged with UserId
// even methods that are in deeper callstacks
System.Diagnostics.Activity.Current?.AddTag("UserId", message.UserId);
// call a third party service 
var result = await _someService.GetStuff(message.UserId, cancellationToken);
if (creditLine == null)
{
    // this warning will automatically get tagged UserId
    _logger.Warning("unable to find data");
    return this.Succeed();
}

Instrument your application

public async Task<Result> RenderEmail<T>(T emailViewModel, string userId, string clientId)
    where T : IAccountEmailViewModel
{
    var emailType = emailViewModel.GetType().ToString();
    using var operation = _telemetryClient.StartOperation<DependencyTelemetry>($"Render email {emailType}");
    operation.Telemetry.Type = "Email";
    operation.Telemetry.Properties["EmailType"] = emailType;
    return await _razorViewToStringRenderer.RenderViewToStringAsync(emailViewModel, clientId);
}

Note this code example uses using declarations which is new to C# 8.0. https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-8.0/using

Enable HTTP/2 on App Services

Reason: There are many performance improvements in HTTP/2 such as header compression and binary formatting in HTTP/2 and “head-of-line” blocking problem.

Enable Always On and configure the always endpoint correctly.

Reason: Enabling Always On avoids dreaded cold start. This can be enabled via ARM template, scripts or manually in the Azure Portal. However the always on endpoint in the Web.Config of the app service also needs to be updated so that your logs don’t flooded with failed requests from always on pings.

This can be done simply by adding the following Web.Config file same folder where your Api’s csproj file is. dotnet publish will automatically include this file in the publish output.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
    <rewrite>
      <rules>
        <rule name="Rewrite AlwaysOn" stopProcessing="true">
          <match url="^$" />
          <conditions>
            <add input="{HTTP_USER_AGENT}" pattern="^AlwaysOn$" />
          </conditions>
          <action type="Rewrite" url="https://{HTTP_HOST}/api/ping?source=alwayson" />
        </rule>
      </rules>
    </rewrite>
    <httpProtocol>
      <customHeaders>
        <remove name="X-Powered-By" />
      </customHeaders>
    </httpProtocol>
  </system.webServer>
</configuration>