Seulgie HanApache Airflow (8) RefactoringWhen building ETL pipelines in Apache Airflow, it’s easy to end up with monolithic tasks that are hard to maintain or debug. In this post…Nov 28
AccentFutureMastering Real-Time Data Pipelines: API Integration with Apache Airflow and PostgreSQLIn today’s fast-paced data ecosystem, the ability to process real-time data efficiently is key to driving informed decision-making and…Sep 25
Seulgie HanApache Airflow (5) Data PartitioningData Partitioning is essential for optimizing ETL pipelines, especially in Apache Airflow, where handling large datasets requires efficient…Nov 16Nov 16
Murat ÖzcanUsing Default Args in Airflow: Make Your Workflows More EfficientHello, in this article we will examine the Use of Default Args used in airflow. Airflow is a powerful tool for managing data processing and…Feb 162Feb 162
Seulgie HanApache Airflow (4) Scheduling and CatchupWhen working with data pipelines in Apache Airflow, one key strategy for improving efficiency and accuracy is to narrow down the scope of…Nov 13Nov 13
Seulgie HanApache Airflow (8) RefactoringWhen building ETL pipelines in Apache Airflow, it’s easy to end up with monolithic tasks that are hard to maintain or debug. In this post…Nov 28
AccentFutureMastering Real-Time Data Pipelines: API Integration with Apache Airflow and PostgreSQLIn today’s fast-paced data ecosystem, the ability to process real-time data efficiently is key to driving informed decision-making and…Sep 25
Seulgie HanApache Airflow (5) Data PartitioningData Partitioning is essential for optimizing ETL pipelines, especially in Apache Airflow, where handling large datasets requires efficient…Nov 16
Murat ÖzcanUsing Default Args in Airflow: Make Your Workflows More EfficientHello, in this article we will examine the Use of Default Args used in airflow. Airflow is a powerful tool for managing data processing and…Feb 162
Seulgie HanApache Airflow (4) Scheduling and CatchupWhen working with data pipelines in Apache Airflow, one key strategy for improving efficiency and accuracy is to narrow down the scope of…Nov 13
Stephen DillonTesting Apache Airflow DAGs with PytestOne of the amazing aspects of Apache Airflow is how easily it enables testing of DAGs. Below we will test a DAG using Pytest including how…Aug 3
Mahmood HossainData Engineering Using NIFI, Hive, Spark, Kafka and QuicksightIn this blog I will explain a project to create a data pipeline where we will use Apache NIFI to ingest and distribute data into different…Apr 26
