(A High-level Overview of Executors in Apache-Airflow) — Written in collaboration with Hiren Rupchandani Preface Airflow executors are the mechanism that handles the running of tasks. Airflow can only have one executor configured at a time in the airflow.cfg file in the airflow’s root directory. Primarily we can run executors in two modes, locally, and remotely, and they can…

(An Illustration of the Apache-Airflow Fundamentals) — Written in collaboration with Hiren Rupchandani Preface In the previous stories, you learned how to set up the airflow in Windows (using WSL), Ubuntu, and macOS operating systems. It’s finally time to show you how to create your first DAG in airflow! In this story, you go through some essential concepts…

(A Step-by-Step Guide to Setup Apache-Airflow on macOS) — Written in collaboration with Hiren Rupchandani Preface In the previous stories, you observed and learned a high-level overview of Data and Data Pipelines, followed by an Introduction to Apache Airflow. …

(A Step-by-Step Guide to Setup Apache-Airflow on Linux) — Written in collaboration with Hiren Rupchandani Preface Previously, you observed and learned a high-level overview of Data and Data Pipelines, followed by an Introduction to Apache Airflow. Moreover, I also explained the setup of Apache-Airflow on Windows 10 using WSL 2. However, that’s not the stop. …

(A Step-by-Step Guide to Setup Apache-Airflow on Windows 10) — Written in collaboration with Hiren Rupchandani Preface In the previous story, you learned to set up Ubuntu 20.04 on Windows 10 as Linux Subsystem Distribution. In this article, I will walk you through the installation process of Apache Airflow in WSL 2 using a virtual environment. Installation of pip on WSL 2 To set up a virtual…

(A Beginners Guide to Setup Windows Subsystem for Linux 2) — Written in collaboration with Hiren Rupchandani What is WSL? WSL stands for Windows Subsystem for Linux provided by Microsoft. It is a tool that helps us integrate and run any Linux distribution natively (such as Ubuntu) on Windows. …

(A Gentle Introduction to Workflow Management Platform) — Written in collaboration with Hiren Rupchandani Preface In the previous story, I gave an overview of data and data pipelines, their necessity, the dynamic components, and the tools required to build data pipelines. In this story, I will walk you through a popular workflow management framework called Apache Airflow. Let’s dive…

(Introduction & Importance of Managing Data Flows) — Written in collaboration with Hiren Rupchandani Preface In the last decade, we saw a boom in the innovations of big data technology due to the emergence of a copious amount of data. …

(Working of Machine Learning with Graphical User Interface) — Preface Many people believe that it is crucial to have an understanding of coding to execute machine learning models. However, it is not entirely true. You don’t require, and it is not essential to learn coding to develop data solutions for your stakeholders. The only things that matters are logic, research…