2. Running Airflow Locally (in a Python Environment)
My personal notes from the book “Data Pipelines with Apache Airflow” by Bas Harenslak and Julian de Ruiter — Chapter 2, Part 2
Introduction
This series of posts is meant to summarize my learnings from the book by Bas Harenslak and Julian de Ruiter. If you like the content, you can purchase the book on Manning.
Complete list:
Chapter 2:
- 1. Introduction to Airflow
- 2. Running Airflow Locally (in a Python Environment)
- 3. Running Airflow with Docker
- 4. Understanding Airflow User Interface
Running a DAG
In order to run a DAG, you need to have Airflow up and running. You can either install Airflow in your Python environment or use Docker. At least you need to have: a scheduler, a webserver, and a database.
Running Airflow in a Python Environment
The official package is called apache-airflow and can be found here.
Whenever you have a Python project is good practice to create a separate environment in order to avoid conflicts among package…
