Setting up Airflow to run with Docker Swarm’s orchestration

The code snippets to set up a reliable scheduler

Some time back, I posted a story describing how Airflow can be used along with a container orchestration layer (like Docker Swarm) to build a resilient task scheduler. In this story, I want to share the minimalistic code that is required to get that up and running. We will be setting up Airflow to run tasks on Docker Swarm. For readers who are new to Airflow or Docker Swarm, I would recommend having a look at the previous story first.

At the end of this article, you will have a running instance of Airflow with its tasks running automatically on any server (from your pool of servers) with enough resources. If you just want to learn to set up Airflow and run tasks on it, this might still be a good place.

The content

Here are the steps we’ll be following:
1. Installing Docker (shipped with Docker Swarm in-built)
2. Setting up a Docker Swarm cluster
3. Installing Airflow
4. Running your tasks with Airflow on Docker Swarm

So let’s get started.

Installing Docker

This is the only part I won’t get into many details since it is quite OS-dependent and already well documented at https://docs.docker.com/install/#supported-platforms. If you still find any problems installing it, feel free to ask me in the comments or on Stackoverflow.

Once you have (or if you already have) Docker running on your system and it is accessible through the command line, you can jump to the next section.

Setting up a Docker Swarm cluster

From your terminal, run this command — docker swarm init

You should now be able to see one node in your cluster when you run the command given below:

docker node ls

Docker Swarm cluster with 1 node

Add more nodes to the cluster (optional)

Right now we have a Docker Swarm cluster with one node which is enough to see things in action, so you can skip this step.
If you have some more nodes (servers) and want to see the orchestration layer (Docker Swarm) doing some “real” work, you can add more nodes to the Docker Swarm cluster with the command that you would have got as a result of the “init” command ran earlier (for e.g. docker swarm join --token SWMTKN-some-long-string-token-here some-ip-or-hostname-here:2377). Once you add more workers to the Swarm cluster, you will see more nodes returned by the command docker node ls.

Installing Airflow

Since we have Docker installed, we can simply run a Docker image of Airflow and get it up & running. There is also a PostgreSQL instance that we will run to store Airflow’s metadata (many other databases are supported). To do that run the following commands:

mkdir airflow-swarm-operator-blog
cd airflow-swarm-operator-blog/
curl -XGET https://gist.githubusercontent.com/akki/cd89c9caff5bf19454cdf913ceb59c32/raw/085f4f538c2d48e01d7789aa8701f9f00e3a0a81/stack.yml >> stack.yml
docker stack deploy --compose-file stack.yml airflow

Note: Windows users might have to change /var/run/docker.sock:/var/run/docker.sock to //var/run/docker.sock:/var/run/docker.sock in the stack.yml file (created by the above commands) as stated here.

You should see the following output (make sure you don’t see any errors):

Output after deploying Docker stack

To put it in simple words, we just created a Docker stack file and deployed it. This will start 2 Docker services which can be confirmed with the following command:

docker service ls

If everything works as expected, the output should look something like this:

Airflow and Postgres running in Docker Swarm

Also, you should now be able to see Airflow’s beautiful UI at http://localhost:18080.

Scheduling a task (DAG) in Docker Swarm

Now that we have Airflow set up and running, the next step is to run our tasks with Airflow on Docker Swarm. One thing to note over here, if you are new to Docker Swarm, is that in Swarm all containers run as a “service” so whatever task (called DAG in Airflow) you schedule for running will run as a Docker service.

Adding a DAG with DockerSwarmOperator

Since Airflow is running inside a Docker container we need to copy the DAG file to the container to add the DAG to Airflow. Simply follow these steps for that:

1. Download the DAG file by running curl -XGET https://gist.githubusercontent.com/akki/4c95805ce1617e2765fecb73bb98230c/raw/cf58ec0391957d50f5ae548f7f618a58c9395156/example_dockerswarmoperator.py >> example_dockerswarmoperator.py.
2. Run docker ps in your terminal.
3. Look for the container name airflow_podand copy its container id in the previous step’s command (highlighted in the image below).

Result of “docker ps” command when Airflow stack is running

4. Run docker cp example_dockerswarmoperator.py <copy-container-id-here>:/root/airflow/dags/example_dockerswarmoperator.py. Make sure you get no errors.

5. Now refresh the UI again - within a few (typically 3–5) minutes you should see your DAG in the UI. Congratulations — your DAG is now running in Airflow! 👏

DAG added to Airflow

Confirm that your task is running on Docker Swarm

So that was it! You just ran your first Airflow task on Docker Swarm.

But how can we know that for sure? We can confirm this by noting that a new Docker service starts when we trigger our Airflow task. Here’s how to do it:

1. Start (trigger) your task via Airflow UI by clicking the “Trigger DAG” button — the “play” button under the “Links” section.

Trigger Dag button

2. Confirm that your task is running in Airflow i.e. there is one task with state=“running”.

Airflow DAG with a running task (the light green circle means “running”)

3. Go to the terminal and run docker service ls — after a few seconds, you should be able to see a new service running, with its name of the form -> airflow-<8-character-random-string>. (Note that the service runs only for 45 seconds, so after that, your service will terminate and disappear from here).

Airflow task running as Docker service.

THERE YOU GO! Your task ran as a Docker service. If you had added more than one nodes to your Docker Swarm cluster (as told in the optional step previously), you’d see it running on different nodes if one node is short on resources.

Additionally, if you are familiar with Airflow, you might be able to print something from your DAG and check that it gets shown in the Airflow DAG’s logs.

Shutting down everything

If you’d like to stop everything we started, here are the commands to do that:

1. Remove the Docker stack

docker stack rm airflow

2. Make sure there are no services running anymore

docker service ls

3. Leave the Docker Swarm cluster

docker swarm leave --force

4. Optionally, remove the directory created for this blog

cd .. && rm -r airflow-swarm-operator-blog

Conclusion

After running through all these steps, you should be able to run your Airflow jobs in Docker Swarm via the DockerSwarmOperator (see this to know more about it). If you have any questions, please drop them in the comments below and I’ll be happy to help out if I can.

For anyone curious, here is the Dockerfile for akkidx/airflow:blog1 — https://gist.github.com/akki/55eae1e196b05377bd7b3031d9c16bc6.