Application Data Monitoring
What Is It, And Why Do You Need It?
As software applications have evolved and grown more complex, so have the technologies and tools for monitoring them. Today, the term application monitoring is considered synonymous with Application Performance Monitoring (APM), a field that deals with understanding application performance by tracking metrics like latencies and errors, and helping prevent, detect, and resolve performance issues and outages. The field of APM is active and bustling with excitement from the DevOps community.
However, today’s changing landscape of technology and governance is pushing us towards monitoring applications on a much deeper level, by monitoring the data itself that flows through our applications.
It is this changing landscape that got Arjun and I together to start Layer 9. We argue that we need a new field, one we are dubbing Application Data Monitoring (ADM), devoted to understanding and monitoring application data. ADM deserves an entire manifesto of its own, but in brief, we believe it must include three defining characteristics:
- First, ADM needs to be broad in its coverage of the entire lifecycle of data: from the users of the application where data is collected, to the services where it flows through, places where it is used, third parties that access this data, and the persistent stores where data is written to and read from.
- Second, ADM needs to gain a deep semantic understanding of data that goes beyond the traditional metrics of throughput and latency. This includes understanding the schema elements of the data, its ownership, lineage, quality, as well as its risk and sensitivity.
- Third, ADM needs to be automated and real-time, supporting dynamic, ever-evolving, data-driven applications.
A lot of technologies and frameworks today fall short of being an “ADM”. APM lacks the semantic depth needed for data monitoring. Data Cataloging and Data Governance technologies focus primarily on data at rest, and lack understanding of data flowing through applications, and manual approaches today for understanding the flows of data are craving real-time automation.
So why do we need the new field of Application Data Monitoring, and what are the forces that are making it so critical today?
A Push from Three Changes in the Technology Landscape
Microservices : Widespread adoption of microservice architecture has changed the way we think about data. Applications no longer have centralized data stores or schemas. Instead, we have hundreds of autonomous services, if not thousands, each encapsulating its own data, working together to form the application. The majority of enterprises adopting microservices architecture have cited lack of visibility into end-to-end processes as one of the top challenges. The flow of data is what ties everything together, and understanding data flows is the key to gain back end-to-end visibility.
Continuous Deployments : Continuous integration, delivery, and deployment (known as CI/CD), which is becoming the norm today, means that enterprises change their production systems several times a day. Contrast this rapid-fire pace of development against the current manual approach of understanding, documenting, and monitoring data, and you can see how painfully slow, static, obviously outdated, and altogether unfeasible it really is.
Proliferation of SaaS : Most enterprises today make use of a large number of cloud applications and third-party APIs for various parts of their applications like data storage, communication, monitoring, marketing, sales, data enrichment, etc. The proliferation of SaaS means that data crosses several boundaries of ownership, which makes it even more critical for companies today to understand and monitor the flow of their data.
A Pull from Three Governance Needs
Privacy and compliance: The increasingly tightening regulatory landscape, with recent regulations like GDPR and CCPA, is putting a lot of pressure on companies to monitor their usage of data. This includes monitoring what data is being collected from the users, how it is used within enterprises, what is shared with third parties, and providing transparency and control over data to their end users. This is especially challenging for modern technology stacks as we outlined above, and begs for an Application Data Monitoring solution.
Security. You only have to take a cursory look around at the alarming frequency of data breaches to see that data security matters more than ever. This is especially true for modern technology stacks, with decentralized data storage and with data flowing through myriad internal services as well as externally to third parties, and the services controlling the access and flow of data are themselves changing multiple types a day. This new and constantly fluctuating reality calls for an automated, real-time, deep understanding of data, its sensitivity, and its risk as it flows through our applications.
Engineering productivity. Modern technology stacks were designed for engineering organizations to scale, and to be agile and autonomous. However, they have brought their own set of new challenges to the engineering organizations making this shift. These challenges include understanding dependencies for their services, making changes, testing applications for correct behavior, trusting the quality of data that flows into their services, figuring out ownership of data and services, and documenting their ever-evolving system architectures, to name a few. In the complex and fragmented world of microservices, data is the life force that connects everything, and we need powerful tools for engineers to understand data flows and overcome the visibility challenges posed by modern stacks.
The confluence of multiple forces that I’ve described here, both technological and governance-related, are poised to make Application Data Monitoring a critical need. Today’s tools to understand data flows are underwhelmingly outdated, manual, and static, providing neither complete nor accurate visibility that’s sorely needed by modern technology stacks. ADM can uniquely serve today’s applications by a deep understanding of the flow of data, enabling modern workflows, and providing an umbrella of automation for more complex governance and engineering needs.
I believe ADM will not be a single technology, product, or use case, but rather a whole suite of interoperable technologies, common frameworks, open standards and APIs, and reference architectures, with an enthusiastic community behind it. At Layer 9, we have taken our first step into the exciting world of Application Data Monitoring, and I am thrilled to be part of this journey. Follow our Application Data Monitoring blog on Medium or on Twitter @layer9ai.
