The Case for Data Observability

Adopting a microservices architecture brings several sets of related challenges to an engineering organization. Few parts of the organization are unaffected but developers and devops teams carry the brunt of the burden.

Developers are working hard to ensure that the services that they create achieve key objectives ranging from performance requirements to functional reliability. Simply understanding all the services that are relevant for their team can take significant effort today.

Many developers experience the pain of rolling out a simple change that ends up causing breakage for downstream services. Service documentation is rarely up-to-date and getting basic information about an endpoint or its schema requires a reliance on tribal knowledge. Most engineers have had the painful experience and know all too well how easily one can log sensitive information in plaintext without realizing it… until an audit flags the oversight. Add to this the inevitability of service failures that result from calls to services with broken contracts that take hours and sometimes days to diagnose and fix.

At the same time, devops teams are faced with a complex production environment with dozens, if not hundreds, and sometimes even thousands, of services. Detecting and resolving service issues is a complex proposition.

There are different sources for monitoring metrics, determining who owns a service, and understanding the most recent changes that were made. And, that is just to get started. Going from observed errors back to the underlying root causes today requires switching between dashboards and alerting tools, navigating the organization, a heavy reliance on tribal knowledge and a laborious review of logs and service data.

Errors that affect performance metrics often don’t even tell the full story. Bad data such as a service erroneously producing the same location data for every new signup may never even affect performance and can remain undetected until the business starts to suffer, say by showing information about the wrong location or delivering product to the wrong address, by which time it is often too late to reverse or even mitigate the damage.

Monitoring systems might give provide a sense of which services are up or down but they are incapable of detecting a growing data problem — whether it is silent data corruption that is feeding garbage into the business or minor data type and precision changes that spawn errors in other parts of the system. The modern engineering organization needs a better approach to address the complexity that microservices can bring.

At Layer 9, our objective is to solve the growing data observability challenge as microservices become the de facto standard for building software. We have developed the world’s first data observability platform to shine a light on the data layer and detect data quality issues, schema changes and more in realtime, before they cause headaches for your whole team.

—

Image credit: Photo by Brett Jordan on Unsplash

If you’re interested in learning more, please drop us a note via layer9.ai or follow us on Twitter @layer9ai or on LinkedIn @ Layer 9 AI