Airflow Development Environment with IntelliJ

Lakshay Gupta

Published in

Simpplr Technology

3 min readApr 9, 2024

Introduction

In this tutorial we will learn how to setup apache airflow workspace using IntelliJ.

Prerequisite

IntelliJ Community Edition. If not installed, download here

What is Apache Airflow

A tool to orchestrate jobs for you. These jobs are isolated tasks and airflow helps in orchestrating these tasks as Directed Acyclic Graphs(DAG). Official documentation of Airflow can be found here.

Getting started

Step 1: Open IntelliJ and click on ‘New Project’

Step 2: Fill project name, set location and create python virtualenv using IntelliJ

Step 3: Click on terminal tab at the botton and validate if virtual environment is activated or not. If not activated, type below command to activate and press enter.

source venv/bin/activate

Step 4: Create a new directory from terminal. Your project structure will look like below screenshot.

mkdir airflow

Step 5: Run below command to setup and install airflow in virtual environment. Replace YOUR_USER_NAME with your actual username.

export PYTHON_VERSION="$(python - version | cut -d " " -f 2 | cut -d "." -f 1-2)" && export AIRFLOW_HOME=/Users/<YOUR_USER_NAME>/Medium/AirflowWithIntelliJ/airflow && AIRFLOW_VERSION=2.9.0 && CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt" && pip3 install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

Step 6: Run command airflow standalone in terminal and wait till you see below message in terminal logs.

airflow standalone

Final Step: Open airflow at localhost:8080 and enter username as ‘admin’ and password will be shown in the terminal. Password is also stored in airflow folder we created in Step 4 above.