The New Buzzword in Data Engineering: Zero ETL
What is Zero ETL — Definition, Benefits & Challenges
In the field of Data Engineering, we often hear about the so-called zero ETL approach, but what exactly is this?
Definition
The Zero ETL approach is a method for building data pipelines that aims to eliminate the need for traditional extraction, transformation, and loading (ETL) processes and the tools used to perform them. This approach is based on the idea that data should be stored and processed or even just analyzed within the source system e.g. with SQL in its original format without the need for complex data transformation or movement.
Benefits
At the end of the day, it means that modern cloud-based Data Warehouses, Data Lakes or even Data Lakehouses use the integrated services of the large cloud providers to analyze data directly from other sources. So rather than filtering data from SQL or NoSQL databases, processing and then putting it into your Data Lake or Data Warehouse, etc. two times, one can just easily gain access to the data directly (often simply via SQL). This has several advantages, like:
- Less effort for building up data pipelines, especially less effort if you have previously programmed them.
