When running a lot of microservices, a lot of things happen! You need observability to know what’s happening, the duration of it, and how often. In Danske Bank, we’re running a mobile bank across 5+ markets with more than a million users. This is a lot or a little depending on your point of view. Nonetheless, we’re 100+ developers developing (or maintaining) 200–300 microservices.
We try to preach a “you build it, you run it” mindset, which means we need to give developers great and easy insights into what they’re building.
In this post, I’ll show you how to instrument your C# code for observability.
What do we do?
We’ve migrated our entire mobile banking application into microservice. We’re running entirely on dotnet core. Our stack is as follows:
- dotnet core
As you can see from the above, there’s no metrics store.
We’re going to use Prometheus. This is a de-facto standard in the industry and we’re not going to challenge that with this series.
The 3 pillars of observability
I’m a big fan of the idea of the 3 pillars of observability. To recap they are logging (which we have in place), metrics, and tracing (which we have in place as well). As we’re, largely, missing metrics from our microservices, this is where I’ll focus.
What we want is for any external dependency to be instrumented. This means that every SQL query, RabbitMQ publish, HTTP call, etc. must be instrumented.
There’s a number of ways of achieving this. As we’re already using custom Nuget packages extensively, we’ve decided to provide packages for interacting with dependencies. Both to help teams using the right tool for the job but, more importantly, to ensure a uniform and instrumented usage.
The following series will contain a walkthrough of how we implemented an injectable Prometheus client for C#. Furthermore, it will cover how we implemented instrumented clients for RabbitMQ, SQL and HTTP calls in a simple yet effective manner.
A DI’able Prometheus client
Disclaimer :): I’m a huge fan of dependency injection. So naturally, I want to be able to inject a client for metrics.
We’re going to build upon the prometheus-net library which exposes a static class for observing or measuring operation.
We will wrap this in something which has a nice interface and is mockable and injectable to improve testability.
So how should our interface look? Something like this
Ideally, this is what we want. However, we also want to know what is actually the metric.
So let’s add a metricKey
This helps a lot. However, I want to be able to group metrics. Say I want to know what state our SQL servers are in. So let’s add a (for lack of a better word) metric group.
Now it’s getting a bit crowded. We ended up creating a MetricKey class that contained this data as well as some additional key/value pairs for adding custom labels.
and our interface will then look like this:
Implementation wise, the concept of adding labels when storing the metrics, is a bit awkward, so beware.
As you can see above, we add bounded context and service as well as we’re running a (sort of) domain-driven microservice architecture. Furthermore, we add an operation and a success label denoting the name of the metric key as well as whether or not the operation succeeded.
So, right now we actually have a base for instrumenting our code.
The client is used in the following manner:
Sprinkling fairy dust
In order to make the metrics client even nicer, we added a method that would allow us to start the measuring directly.
The MetricMeasurement (yes, horrible name, but bear with me) is then responsible for invoking the Action (in this case, Store, on lines 19 and 26) whenever it’s finished.
This way, we can use a using statement for the MetricMeasurement and when it’s disposed of, the metric will be stored.
In the example above, the metric will be stored as successful unless explicitly marked as failed.
This extension will not change the functionality but will add a nice API for instrumenting code as we’ll see in the following.
Microsoft SQL Server
We rely quite heavily on Microsoft SQL Server. Usually, we use dapper or Entity Framework to query.
In the following, I’ll describe how we use the MetricsClient to instrument dapper.
This isn’t the definitive guide, but something that works in practice for most use cases.
I’ll show an example of how we support queries and executions. This assumes a minimal understanding of dapper.
What it’ll give you, in addition to instrumentation, is a nice, injectable SQL client.
We’ll create an interface ISqlClient
and create an implementation of this using dapper. Of course, you may need to support more of the features dapper exposes, and then you can just extend the interface.
The SqlClient implements the ISqlClient interface which allows us to decorate it nicely as in the example above.
Again, this isn’t anything we’ve magically come up with, but a practical implementation of code instrumentation using the existing libraries, we’re already using.
We have done this for all of our dependencies so that (over time) all of our I/O will be nicely visible in Prometheus (or Grafana).
We now have a suite of instrumented clients for our common dependencies. This allows developers to easily and uniformly use these dependencies while easily be able to create advanced dashboards to monitor their services.
By Mathias Hermann
Senior Software Architect, Danske Bank
I’m Mathias Hermann, I have a degree in software development and I’ve been employed in Danske Bank since 2016. My role in all of this, is that I’m a Software Architect which is probably the least saying description. What I do is developing software, trying to get or maintain an overview of what we’re doing and where we should be going, and then I consult with teams on new products or projects. Moreover, I’m very passionate about code instrumentation and service observability.