Simplified Apache Airflow Installation on Windows 10 Using WSL

In the world of data management, Apache Airflow shines as a powerful tool. But for Windows 10 users, installing it used to be a challenge. Not anymore! In this guide, we’ll simplify Apache Airflow installation on Windows 10 using Windows Subsystem for Linux (WSL). Let’s explore this seamless integration together. Welcome to an effortless journey of automation and efficiency.

Step 1: Installing Ubuntu on Windows

To begin our journey with Airflow, we first need to set up Ubuntu on Windows 10. Luckily, the Microsoft Store provides a straightforward solution for this step. Let’s start by installing Ubuntu to pave the way for our Airflow adventure.

Microsoft Apps

With Ubuntu successfully installed on your Windows 10 system, the next step involves finalizing the setup. When you launch Ubuntu for the first time, it will prompt you to create a new username and password.

Once you’ve provided a new username and a strong password, your Ubuntu installation will be fully set up and ready to use. At this point, you can close the Windows Subsystem for Linux (WSL) window for the moment.

Step 2: Enhancing Your Terminal Experience

Let’s face it: the Windows command line might lack the charm and versatility of its native Ubuntu counterpart. To bridge this gap and elevate your terminal experience, head over to the Microsoft Store. There, you can download the Windows Terminal, a feature-rich app designed to make your command-line interactions far more enjoyable and efficient.

Microsoft Apps

Now that you have the Windows Terminal installed, it’s time to optimize your setup for Ubuntu. Follow these simple steps to ensure a smooth experience:

1. Open Windows Terminal: Launch the Windows Terminal application on your system.
2. Access Settings: Click on the downward arrow icon in the title bar of Windows Terminal and select “Settings” from the dropdown menu.
3. Set Default Profile: In the Settings menu, navigate to the “Profiles” section. Here, find the profile for Ubuntu/Linux.
4. Set as Default: Click on the Ubuntu/Linux profile and look for the “Make default profile” option. Click on it to set Ubuntu as your default terminal profile.

By setting Linux as your default profile, you ensure that every time you open the Windows Terminal, you’re greeted with the familiar Ubuntu command line interface.

Step 3: Installing Dependencies for Airflow

With your Ubuntu environment ready and Windows Terminal configured, it’s time to prepare the groundwork for Airflow. We’ll start by installing essential dependencies — pip and virtualenv – within your Windows Subsystem for Linux (WSL). These tools are vital for managing packages and creating isolated environments for your Airflow projects.

Installing Pip:

In your Ubuntu terminal, execute the following command to install pip:

$ sudo apt update 
$ sudo apt install python3-pip

This will ensure you have the latest version of pip installed on your Ubuntu system, enabling seamless package management.

Installing Virtualenv:

Next, install virtualenv to create a controlled environment for Airflow:

$ sudo pip3 install virtualenv

By setting up virtualenv, you create a sandboxed environment specifically tailored for Airflow, allowing for efficient management of packages and dependencies.

Step 4: Setting Up Airflow in a Virtual Environment

Now that you have pip and virtualenv installed, it's time to create a dedicated workspace for Airflow. Follow these steps to create a directory, set up a virtual environment, and download Airflow:

1. Create a Directory and Virtual Environment:

In your Ubuntu terminal, navigate to your desired location (for example, your home directory) and create a new directory named “airflow”:

$ mkdir airflow
$ cd airflow
$ virtualenv airflow-env

This will create a virtual environment called “airflow-env” within the “airflow” directory.

2. Activate the Virtual Environment:

To activate the virtual environment, use the following command:

$ source airflow-env/bin/activate

Your terminal prompt should change, indicating that you are now working within the virtual environment.

3. Install Airflow:

With the virtual environment active, Install Airflow using the following command:

$ sudo apt install apache-airflow

Ubuntu will fetch and install Airflow and its dependencies within your virtual environment.

Step 5: Configuring Airflow and Creating Users

Now that Airflow is installed in your virtual environment, let’s set up the necessary directories, configure environment variables, and create user accounts for seamless operation.

1. Navigate to the Airflow Directory:

$ cd ~/airflow

2. Create a DAGs Directory:

Inside the Airflow directory, create a folder named “dags” to store your DAGs (Directed Acyclic Graphs):

$ mkdir dags

3. Add Airflow to the PATH:

To ensure easy access to Airflow commands, add the Airflow home path to your .bashrc file. Open the .bashrc file using a text editor:

$ nano ~/.bashrc

Add the following line at the end of the file:

# Airflow Path
export AIRFLOW_HOME=~/airflow

Save and close the file. To apply the changes, either close and reopen the terminal or run:

$ source ~/.bashrc

4. Initialize the Airflow Database:

Initialize the Airflow database to set up the necessary database schema:

$ airflow db init

5. Create Admin User:

Create an admin user for Airflow. Replace the placeholders with your desired values:

$ airflow users create --username admin --password your_password --firstname your_first_name --lastname your_last_name --role Admin --email your_email@domain.com

Congratulations! You’ve configured Airflow, created a DAGs directory, added Airflow to your PATH, initialized the database, and set up an admin user. Your Apache Airflow setup is now ready to help you automate your tasks effectively and efficiently.

Step 6: Starting Airflow Scheduler and Web Server

With the setup and user creation completed, it’s time to start the Apache Airflow components and access the user interface.

1. Start the Airflow Scheduler:

In your terminal where your virtual environment is activated, start the Airflow scheduler:

$ airflow scheduler

Leave this terminal window running. The scheduler will continuously monitor your DAGs for any tasks that need to be executed.

2. Open a New Terminal Tab and Activate the Virtual Environment:

Open a new terminal tab and navigate to your Airflow project directory. Activate the virtual environment:

$ source airflow-env/bin/activate

3. Start the Airflow Web Server:

In the newly opened terminal tab, start the Airflow web server:

$ airflow webserver

The web server will launch, and you can access the Airflow UI by navigating to localhost:8080 in your web browser.

Congratulations! Airflow is up and running. You can now access the Airflow web interface, log in with your admin credentials, and start creating and managing your workflows through the user-friendly interface.

Step 7: Customizing Airflow: Removing Pre-Built DAGs

After signing into the Airflow web interface, you might find the pre-built DAGs a bit overwhelming. If you prefer a clean slate to start your workflow management journey, follow these steps to remove the pre-built DAGs from your Airflow environment.

1. Navigate to the Airflow Configuration File:

In your terminal, navigate to your Airflow project directory and open the airflow.cfg configuration file using the nano text editor:

$ cd ~/airflow
$ nano airflow.cfg

2. Modify the load_examples Parameter:

Inside the airflow.cfg file, locate the load_examples parameter. By default, it is set to True. Change it to False:

$ load_examples = False

3. Save and Exit:

Save your changes in nano by pressing Ctrl + O, then press Enter to confirm. To exit nano, press Ctrl + X.

4. Restart Airflow Web Server:

After modifying the configuration, restart the Airflow web server for the changes to take effect:

$ airflow webserver -p 8080

Now, when you access the Airflow web interface at localhost:8080, you'll enjoy a clutter-free environment, ready for you to create your own customized workflows.

Conclusion:

In this guide, we’ve streamlined the process of setting up Apache Airflow on Windows 10 using Windows Subsystem for Linux (WSL). I trust these steps have made the installation journey more accessible for you.

Your feedback and questions are invaluable. If you’ve found this guide helpful or have any insights to share, please don’t hesitate to leave a comment below. Whether you’re an Airflow enthusiast or a beginner taking your first steps, your input can make a significant difference for fellow readers.

Thank you for investing your time in this guide. I eagerly await your thoughts and experiences. Let’s keep the conversation going!