Why Apache Iceberg is heralding a new era of change in Data Engineering
“Bring Your Own Storage (BYOS)” has never been cooler
About me
I’m Hugo Lu — I started my career working in M&A in London before moving to JUUL and falling into data engineering. I headed up the Data function at London-based Scale-up Codat. I’m now CEO at Orchestra, which is a data release pipeline tool that helps Data Teams release data into production reliably and efficiently 🚀
️️⭐️ Also check out our Substack and our internal blog ⭐️
Introduction
For many years, compute and storage were intrinsically linked. If you wanted to own a computer (in the context of business computing), you would need to pick from a menu of computers with differing levels of RAM (compute) and storage.
This bled into software.
Data warehouses typically offered this pricing model. This created tension. What if I had spiky computational requirements? What if storage could change over time?
Snowflake was billed as the first truly elastic data warehouse, as it was designed so both storage and compute could scale infinitely.
