The Future of Observability

Bogomil Balkansky
Oct 13, 2020 · 6 min read

One of the luxuries of my job as a VC is the opportunity to meet exceptional people and to reconnect with old friends. I recently met with a friend who is an engineering leader at one of the high-flying tech companies in San Francisco, and I asked them one simple question: “What’s difficult in your world?” Without skipping a beat their answer was “Observability is a s*** show.” (“Observability” is a set of tools for tracking the health of software environments, and for troubleshooting when things go wrong.)

These words fell on eager ears; I have been tracking the observability space for a few years. Some of my good friends from VMware built SignalFx, and I was an angel investor at Omnition; both companies got acquired by Splunk last year. And at Sequoia we have partnered with Sumo Logic, Lightstep and Wavefront (VMWare).

The bear view on observability is that it will be a quiet space for some time following the recent wave of value creation (Datadog going public, acquisitions by Splunk, Datadog, VMware and others). But I beg to differ. I believe observability is an evergreen area whose collective revenue will only grow over time as subsequent waves of value creation emerge. There are two drivers behind my hypothesis:

  1. Every company has become a software company, and observability is how you keep that software on track. The former CEO of VMware — the incomparable Paul Maritz — used to say that stuff only gets into IT, it never gets out. IT environments are becoming ever more complex, which means that more stuff needs to be monitored.
  2. Despite all the tools available, troubleshooting is incredibly hard. You will never meet an SRE or DevOps person who will tell you that troubleshooting is easy and under control. This means that a company with a better approach in some aspect of observability has a chance of building a sizable business.

Every platform shift — whether at the hardware, infrastructure software or application layer — creates the need and the opportunity to rethink, and likely merge, the observability pillars of monitoring, logging,and tracing. The shifts going on today have already given rise to new types of observability tools:

The shift to microservices created the conditions for companies like SignalFx, Omnition and Lightstep that specialize in monitoring the new generation of cloud-native applications based on microservices. The recent emergence and success of OpenTelemetry makes it easier for instrument apps to collect more data. The combination of an ever-growing number of microservices and the ease of instrumenting apps to collect more data causes the amount of observability data to explode, which requires new ways of storing it and processing it. New time series databases like M3, TimescaleDB and InfluxDB are some of the new ways to tackle that challenge. Cribl is a company pioneering another innovative approach to dealing with the deluge of observability data: the observability pipeline that decouples the collection of data from the ingestion of data into various destinations.

This approach is similar to how centralized the collection and transformation of all customer data from any source so that it can be consumed by any destination — be it another operational or analytical system. Cribl is collecting, filtering and enriching data from the various sources, and making intelligent judgments where data is best stored depending on its value.

My crystal ball says that the most interesting next-generation opportunities in observability will be in one of these areas:

1/ Deep integration with CI/CD

Some observability tools already provide information on how code-merge events correlate with application performance. We partnered with Lightstep (who is very active in this area) and much of their future roadmap centers on visibility into software deployments. The next-gen evolution of this approach will be to identify which code exactly was problematic. Meanwhile, GitHub and others are working on semantic code understanding. These efforts are currently focused on code search and code security, but if and when semantic code understanding matures, it will open very powerful possibilities to understand which particular piece of newly merged code caused the regression or performance degradation. That development would get us closer to finding the proverbial needle in the haystack, or very quickly identify what needs to be fixed when something has broken.

2/ Data observability

3/ End-to-end unification

The observability tool that everyone needs in this scenario starts with the business-level metrics (from whatever BI dashboard sits on the CE/CFO screen), and traces how these metrics are affected by application code and the various tiers of infrastructure.

One of the foundational changes that needs to take place in order to enable end-to-end observability is to unify the observability data and the business data. Currently these are two completely separate worlds: the observability data lives in Splunk, Sumo Logic, or M3, while the business data lives in a data warehouse on prem (Teradata) or in the cloud (Snowflake, Redshift, BigQuery). As a result, it is very hard to tell if a change in a business metric was caused by a technical problem or by some shift in the business.

The next big challenge would be to unify the separate worlds of observability and business analytics tools — at the end of the day they are all about slicing and dicing data in a visual way in order to understand it. I hope in the coming years someone will take on the daring challenge of unifying these two domains.

Sequoia Capital Publication

From idea to IPO and beyond, Sequoia helps the daring build…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store