MDS is dead, Embracing the Post-MDS Era
In a recent statement, Tristan Handy, the CEO of dbt Labs, made a thought-provoking declaration about the term “Modern Data Stack” (MDS). According to Handy, over the span of approximately seven years, the term has undergone a transformative journey from a descriptive technical term to a meme-like market trend, eventually evolving into an entire ecosystem. However, Handy believes that today, the term has lost its utility in all of these roles. In fact, he is personally taking a firm stand against using the term “modern data stack” and suggests that others should follow suit.
Against this backdrop, this article we will provide an in-depth exploration of the evolution of data stacks. While acknowledging the significant traction MDS has gained as a marketing buzzword, we will delve into the historical context to shed light on its origins. Moreover, we will offer insights into what lies ahead in this exciting era of Post-MDS. Whether you find yourself at a critical crossroads in decision-making within this domain or simply seek a deeper understanding, this article is designed to assist you on your journey.
The Past: A Legacy of Custom-Built Solutions
In the past, data management heavily relied on custom code tailored specifically for platforms like IBM System 360, Netezza, or Teradata. As the need for accessible solutions grew, these companies responded by introducing new products such as Informatica, DataStage, and various low-code tools. These offerings aimed to address the challenges associated with data organization, migration, and accessibility.
The Present: The Rise of Cloud Storage and Warehousing
The present era witnesses the massification of cloud storage and warehousing, revolutionizing the data landscape. Native-cloud tools have inundated the market, empowering organizations to navigate this new environment with greater ease and efficiency. Processes that once demanded months of effort from an entire team can now be accomplished by a single individual. However, this progress comes at the cost of increasing complexity. The ever-expanding ecosystem of tools, ranging from open-source to partially proprietary and cloud-native solutions, presents new challenges.
The Future: Consolidation and Integration
Looking ahead, we anticipate a future where data tools consolidate, alleviating the stress and complexity associated with compatibility issues. Companies are likely to acquire one another, leading to the evolution of data platforms with comprehensive vertical and horizontal integration across the data space. Initially, we observed the integration of cloud infrastructure with software, resulting in the emergence of Software-as-a-Service (SaaS) products. Subsequently, a concise ecosystem of tools encompassing databases, ingestion mechanisms, data transformation capabilities, governance frameworks, analytics, and AI/ML functionalities is expected to emerge.
Jon Hsieh, in his insightful presentation, draws parallels between the evolution of data stacks and architectural styles. He likens this transformation to the architectural progression from classical to modern and, ultimately, post-modern styles. Classical systems paved the way for subsequent advancements.
The Changing Role of Data Engineers
As data stacks evolve, so do the roles and practices of data engineers. The journey began with software engineers who gradually specialized in Extract, Transform, Load (ETL) processes. However, the current landscape has given rise to a new role known as Analytical Engineers, reflecting the growing emphasis on data analytics and insights generation.
Conclusion
The Modern Data Stack era has reshaped the way organizations manage and leverage data. As we transition towards the Post-MDS era, the future holds the promise of consolidated tools, simplified compatibility, and comprehensive integration across the data space. By understanding the historical context and anticipating future trends, businesses can make informed decisions that propel them towards data-driven success.
Stay tuned for more updates on the fascinating world of data stacks and the exciting possibilities that lie ahead in the Post-MDS era.
