How to Better Understand your Applications Real-Time Value

Through domain observability, easy-to-use instrumentation, and improved error handling

Matching Engine Example

Software engineers progressing through the development lifecycle can find it increasingly difficult to quickly asses the overall performance of an application. Furthermore, there’s a wide range of disciplines that may be interested in an applications overall value: product owner, developer, CTO, software architect, Director of Product, etc. Which begs the question…

How can developers, product managers, business analysts, and senior executives feel confident they ALL have the tools to accurately evaluate the real-time value of a piece of software?

Firstly, you can assume that any tactics deployed to increase accountability within a piece of software must be as accessible as possible. This increases the probability of adaptation, implementation, and adds more inherent accountability in the system. There are three concepts that can be implemented to increase overall accountability;

• domain-oriented observability pattern
• improved side-effect handling
• visually efficient instrumentation

Detailed by Pete Hodgson in Domain-Oriented Observability , observability can mean many things from low level hardware metrics, to business domain specific performance indicators. At its core domain-oriented observability aims to clean up instrumentation (i.e. logging, metrics, analytics, etc.) within your code base. This pattern will help developers to better isolate buisness logic from distracting instrumentation. After your team becomes familar with the core concepts of this pattern, you can implement this pattern in the following manner:

1. Create your domain probe → Assign this task to a mid/senior level developer
2. Choose a simple domain class (e.g. OrderBook)→ Junior pair with mid/senior
3. Migrate all implementation from the class chosen in Step 2 to your domain probe → Junior pair with mid/senior
4. Repeat steps 2–3 until all instrumentation in your code base has been migrated out of the domain logic classes, and into your domain probe(s) → Junior or new team member

Implementing this pattern in your application is an excellent way for junior, and/or new teammates to learn about the software system. Additionally, the risk of negative impact to your codebase is quite low considering mostly logging and/or metric logic is being refactored. It also helps to establish a precedent for developing clean more readable code. Below is an example of a function to add a buy order to the order book within a matching engine, prior to being refactored.

BEFORE Domain Observability Implantation

Now lets see what happens when we apply the principles of the domain observabitliy pattern to our function. You’ll notice a reduction in lines of code, as well as a more readable function now that you’re not distracted by wordy instrumentation.

AFTER Domain Observability Implementation

This sets developers up to begin some more involved code clean up, in particular the “side-effects” within your application. Stealing from the Arrow.kt definition…

side effect is an externally observable effect a function performs in addition to returning a value.

Side effects like errors, and exceptions can impede the readability and maintainability leading to a confusing stack trace and more time spent debugging. Arrow.kt is a Kotlin library and will be used to demonstrate this concept. Enter the Either functionality in the Arrow.kt Core library. The Either class, can be used in a lightweight manner to handle exceptions as described here. You’ll see below that the function has been further collapsed to better handle an error side effect.

AFTER Arrow.kt Either Implementation

It’s worth noting that this particular library is specific to the Kotlin language, however the approach of better error handling can be more easily applied once the domain obersvability pattern has been implemented.

Last, but certainly not least its vital that you have easy-to-interpret instrumentation. Much like Homer Simpson’s work environment at the nuclear plant, your developers and key stake holders need a place to go where the health and performance of the application can be easily assesed.

Most often this takes the form of a dashboard. Grafana is one of many products in the marketplace that enable this functionality. Furthermore, wiring up a dashboard to an application should be much more straightforward now that all instrumention within the codebase has been abstracted out of business logic and better organized. Below is an example of a basic dashboard that can help better articulate what is happening in the example ‘Matching Engine’ application.

Order Book — Grafana Instrumentation Dashboard

This visualization component enables more accountability. It clearly depicts real-time performance metrics within the application, and can be easily adapted to monitor new application functionality.

Admittedly, there is nothing revolutionary about these patterns. However, putting these concepts into practice can introduces some unforeseen challenges. The approach outlined above is an example of how to tackle some of the low hanging fruit when it comes to code cleaniness. When coupled with common best practices in software development (i.e. pair programming, or mobbbing) it is well suited for software teams with a varying degree of skillsets and experience. Implementing these patterns early in your development lifecycle will yield compounding benefits when it comes to maintaining and growing your application.