VALIDIO
Published in

VALIDIO

5 Data Trends in 2022

In 2021, we saw quite an acceleration of the buzz around the rise of the Modern Data Stack. We now have a tsunami of newsletters, influencers, investors, dedicated websites, conferences, and events evangelizing it. The concept around the Modern Data Stack (albeit still in its early innings) is tightly connected with the explosion of data tools in the cloud. The cloud comes with a new model of infrastructure that will help us build these data stacks fast, programmatically, and on-demand, using cloud-native technologies like Kubernetes, infrastructure as code like Terraform and cloud best practices of DevOps. So, infrastructure becomes a critical factor in building and implementing a Modern Data Stack.

1. The rise of the Analytics Engineer

LinkedIn job posting data as of 1st of December 2021

2. The data warehouse vs data lakehouse war intensifies (and lines get increasingly blurred)

“In the vast majority of accounts that we are in, we co-exist with Snowflake — the overlap in accounts is massive… What we’ve seen is that more and more people now feel like they can actually use the data that they have in the data lake with us for data warehousing workloads. And those might have been workloads that otherwise would have gone to Snowflake.”

The evolution of data storage and warehouses

“I think Snowflake will be very successful, and I think Databricks will be very successful… You will also see other ones pop up in the top, I’m sure, over the next three to four years. It’s just such a big market and it makes sense that lots of people would focus on going after it.”

Data Warehouse vs Data Lake vs Data Lakehouse by Striim

3. Real-time streaming pipelines and operational analytics will continue to push through

“If you don’t have a real-time streaming system, you have to deal with things like, okay, so data arrives every day. I’m going to take it in here. I’m going to add it over there. Well, how do I reconcile? What if some of that data is late? I need to join two tables, but that table is not here. So, maybe I’ll wait a little bit, and I’ll rerun it again.” — Ali Ghodsi on a16z

“Cloud data warehouses were designed to support BI use cases, which amount to large queries that scan entire tables and aggregate the results. This is ideal for historical data analysis, but less so for the “what is happening now?” class of queries that are becoming increasingly popular to drive real-time decision-making. This is what operational analytics refers to. Examples include in-app personalization, churn prediction, inventory forecasting, and fraud-detection. Relative to BI, operational analytics queries join many disparate sources of data together, require real-time data ingestion and query performance, and must be able to process many queries concurrently.”

The stages of data pipeline maturity by Chris Riccomini

4. The rise of Cloud Marketplaces for Modern Data Stack adoption

  • More than 45% of Forbes The Cloud 100 companies actively use Cloud Marketplaces as distribution channels for their software.
  • The committed enterprise spend flowing through the three major Cloud Providers exceeds $250 billion per year — and that number is climbing fast.
  • For 2021 alone, independent software providers generated over $3 billion in revenue through the Cloud Marketplaces, based on Bessemer predictions. Bessemer expects that to grow by a multiple of 10 in the coming years.
  • Forrester had projected that 17% of the $13 trillion global B2B spend would flow through e-commerce and Marketplaces by 2023 — but that number might have been reached already in 2021.
  • A 2020 Tackle survey found that 70% of software providers said they have increased focus and investment in Marketplaces as a go-to-market channel due to COVID-19.
The win-win of Cloud Marketplaces

5. Harmonization and consistency of terminology around the Modern Data Stack and data quality

The data quality category in the 2020 MAD Landscape vs the 2021 Landscape by Matt Turck

In summary

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Oliver Molander

Co-founder at Validio and early-stage tech investor at J12 Ventures. Preaching about the realities & possibilities of Data & ML.