Why Cloud Composer Should Be The Only Way You Run AirFlow
After multiple false starts and hours of frustration attempting local and VM installations, I’ve determined the easiest way to run AirFlow is on GCP.
Apache AirFlow is an increasingly in-demand skill for data engineers, but wow it is difficult to install and run, let alone compose and schedule your first direct acyclic graphs (DAGs). In the next few minutes I’ll share why running AirFlow locally is so complex and why Google’s Cloud Composer takes the stress off of your system and off of your mind.
Note: There are already some fantastic tutorials on installing AirFlow in nearly every kind of environment on Medium and other platforms. Therefore, the purpose of this write-up is to compare and contrast the various environments compatible with AirFlow.
Local AirFlow Installation
Last summer, when I became interested in AirFlow, I first attempted to install the Python package on my laptop, a MacBook Air, in VS Code.
Simple enough?
What followed was days of sifting through Medium articles, StackOverflow answers and haphazardly-composed Google Searches like ‘AirFlow not running why.’ The issue wasn’t the size of…
