Member-only story
Featured
Is Apache Airflow Due for Replacement? The First Impression Of mage-ai
Introduction to mage-ai as the alternative to Apache Airflow for data engineers
Apache Airflow became the top project in Apache Foundation in January 2019. It is one of the most popular projects for data engineers, and many companies list Airflow as one of the requirements for hiring. Data projects iterate quickly, and some projects have been replaced rapidly. Airflow has been widespread for years. Is Apache Airflow due for a replacement? mage-ai is the new ETL tool for data engineers to check out as a substitution. I have taken a first impression of mage-ai and will share my thoughts.
The Data Orchestration War
Data orchestration is highly competitive, primarily because old technology has yet to be fully retired and the increasing arrival of new open-source players. Spotify's Luigi vs. Apache Airflow is forever debatable in the data engineering world. Apache Hop and Apache Nifi have a web-based user interface to crack this problem for users who prefer to rely more on UI as part of the data pipeline development.
Prefect open-sourced project with an Apache 2.0 license after two years of development in 2019. Prefect is new to the data engineering stack, but it learned from years of…
