Upgrading Apache Airflow Locally

TL;DR

Run the following commands to upgrade your Apache Airflow version and restart your Airflow server:

pip install - upgrade apache-airflow==<version number>
airflow db migrate
reboot

Backstory

Recently, I started noticing issues with an Airflow DAG or two where permissions I’d enabled for a certain role were being deleted, periodically.

As any perplexed data engineer would do, I went to Google and came across this GitHub issue which happened to be affecting the exact version I was on, apache-airflow==2.6.2.

The solution is to upgrade Airflow to version 2.7.0 or greater. While it should be (and is) a simple process, shockingly, there is very little documentation available.

The official Airflow documentation goes to great lengths to discuss the airflow db migrate process but doesn’t really explain the Python upgrade commands that need to be run ahead of it. Maybe it’s understood that you just follow standard Python/PIP upgrade protocols, but I always worry when it’s that simple…because it never is that easy!

After speaking with a former colleague, I decided I would upgrade to apache-airflow==2.8.2 because he’s on that version, currently, and was also using the same version of SQLALchemy that I was.

After ensuring IT had taken as backup of the server, I pulled the trigger and ran these commands in my Airflow virtual environment:

pip install --upgrade apache-airflow==2.8.2
airflow db migrate

Naturally, after running those commands, I went to my Airflow URL, tested it out, and got a message to the effect of Oops! Something bad happened…

After panicking for a few minutes, I did what any good tech person would do and restarted the server!

And Boom! We were off to the races again!

I hope this helps someone else who runs into this dilemma!