EDTS

Airflow is a powerful open-source tool for orchestrating workflows and managing data pipelines. Running Airflow locally can be an…

Our solution to this problem involves using Pub/Sub notifications and Cloud Functions in the Google Cloud Platform

In this article, we’ll walk through how to build a real-time data pipeline using Apache Kafka, Apache Flink, and PostgreSQL.

Continuous Integration and Continuous Deployment (CI/CD) pipelines are the backbone of application delivery. However, they are also prime…

Managing workflows can be challenging, especially as they grow in complexity. Apache Airflow, an open-source platform for orchestrating…

In the dynamic and competitive realm of the business industry, data serves a critical roles in decision-making. High volume of data and…

In today’s data-driven landscape, staying updated with real-time data changes is critical for applications requiring up-to-the-minute…

One way to utilize or monetize big data is to make insightful and up-to-date reports. Many BI tools can solve this task, but reporting is…

A data engineer needs to able to managing large datasets efficiently, unfortunately i’m not one of them — at least, not at first…

Importance of Data Quality

It’s better to Trigger the Dag Runs by API when the data is received from the client, for the EDTS case it’s when data is received in GCS