ODIN
A coming of age in Observability
The importance of Observability
Every certain time some topics become hot, popular or trendy in the world of IT. Observability has been one of them in recent years. One of the reasons is the increased complexity of the systems that based on cloud are created in decentralised architectures like microservices, serverless, etc. Every team needs to have a crystal clear view to what’s happening on their applications. We need to have the best way to manage the (sometimes) chaotic world of cloud providers, services, API’s, asynchronous communications etc. Everything has changed a lot in the last decade.
But observability is not only about enhancing the ability to look after your deployments. Sometimes we forget why we are building our applications. The final objective is to enable non IT teams to fulfill business demands. And those have also changed a lot over the last decade . Markets have evolved, customers have different communication channels, the world is at one click at their fingertips. And we need to help them on that too.
Paraphrasing the ‘butterfly effect’ : Does the lag of consumption of a queue in Dublin region affects the market sales in USA ? Well, potentially it might . Because Observability of IT and Business elements are related.
What we have now in Observability
The growing and learning path inside adidas about Observability in the last 5 years has been hard, but also rewarding. Looking into the best practices of the community, the Platform department adapts them to the demands of our development teams offering easy ways (most of the times) to start fast with all the main elements in place.
In Observability we have HOLMES, a custom framework that provides a central repository with all our logs and metrics. We have centrally operated ELK to store the logs, Victoria Metrics for the metrics, Instana for tracing and Grafana to display the information.
Now the Development teams can be autonomous, they can create their dashboards, trigger their alerts and analyse their data. But, could all that massive amount of information help Business more directly, more quicker?
Why ODIN.
All this started with the need of having a better observability of our ECOM journey. There are many applications, internal and external. From the creation of the product, to the campaign, to the payments management and ending in the delivery on the customer backyard.
Most of the IT teams involved in that journey, across the globe and with many different applications and systems, are ready to check the insights of what is happening inside their products with all the tooling we provide. But what about business? What about management areas? We wanted to provide a dashboard for their needs.
The needs were clear : simple dashboards, navigable, showing main KPI’s , marking unreached thresholds , able to connect to different systems across our landscape.
Technically Speaking
Another motto we had in mind was: “make the system fully configurable”. We did not want to rely on a development team to create new metrics or extend them. Business should be capable of managing the dashboards.
The backend itself it’s a very simple Spring Java development, deployed in a K8s cluster. The complexity and power relies in the framework for the business users. For this purpose the backend is feed by configuration files via k8s feature Config Maps. With only configuration the business users can:
- Create new Dashboards or evolve them.
- Create new metrics or adjust them
- Create hierarchies in the Dashboards (fe, different dashboards for different markets)
- Create new Source Systems to connect (fe, a new API)
In order to enhance performance and protect the source systems, the framework includes several levels of cache to protect them from similar queries or requests, and a historical storage of metrics both raw and calculated.
We decided to use GraphQL as the API technology in this case. GraphQL provides many features but two in particular that we wanted to explore:
- Subscribe to the API: A main requirement was to show the information as soon as it was generated. The backend calculates the status of the metrics from the dashboards and returns a very complete response model every few minutes. Once subscribed the UI only needs to wait for the next update to render it.
- Selection of fields on the model : The application is equally used in a laptop, mobile or tablet. At some point we thought even creating an app. We wanted to give the UIs the capacity to select only the data needed to render the dashboard, minimising data transfer. On the other hand the structure is quite complex, but GraphQL contract provides a clean contract to manage that model selector.
There is a third functionality on GraphQL that caught our attention , the possibility of execute queries directly over the model. We are not using that at this point but we are aware that it could be a helpful feature in future.
What do you need to be in ODIN
At this moment the framework is alive and running. And open to include new dashboards to help other teams, from Business or from IT.
The challenge with observability in general is that you need to know what to observe. The only way to establish a threshold is having the KPI available in a system, and a team dedicated to it. In other words: a maturity level of observability is demanded. The Grafana dashboards must be there, to understand the best KPI’s, the alerts must be setup, to trigger your 24/7 if needed, the applications needs an available API to connect to and get the data, etc
The observability of your application needs to start in the development teams, with common systems and guidelines. It needs to start also in your business users. Understand which are the critical KPI’s that you need to look at, understand where your problems are and extract that information from the systems. Only when you have both you can start linking all together to have an eagle view of all your end to end processes.
Our roadmap.
The dashboards are decentrally managed. Any team can have theirs with a minimum development effort (close to zero). The development teams must be working on new features, new integrations, better performance…We have many ideas in the backlog.
- Enable a configuration section to make easier to manage the dashboards: the system is super open, cool. But the users needs to feel comfortable managing it!
- Enable the possibility of getting combined KPI’s from different sources out of just configuration. Those are the most desired calculations every time we ask business!!
- Set dynamic thresholds based on Anomaly Detection : you can’t expect the same demand on all your systems the whole year, there are peaks and valleys.
Conclusion
ODIN covers the initial requirements we had: a simple dashboard to show business KPI’s, responsive UI , configuration able to be managed by non technical people, and extendable to other dashboards and other teams.
It’d be even a candidate for the adidas Developer Portal as an Open Source project in the near future, giving the community the possibility to observe in a simple view a complex ecosystem of areas and applications.
