My Airflow journey began with Udemy courses that I pursued in my free time. However, reality hit me on my first day of a new project when my team lead asked me to examine Airflow logs and explain the cause of a DAG failure. At that moment, I realized that online courses couldn’t fully prepare me for real-world Airflow challenges, so I decided to dive deeper into learning Airflow through hands-on experience and practice.
I’m still a part of the same project, learning something new every single day. My mornings begin with reviewing Grafana to identify Composer errors across various GCP projects. The number of errors varies, ranging from zero to over 1,000 alerts after weekends. Identifying patterns and understanding the root of issues has become one of my key strengths. I’m grateful to have an amazing team that always supports me.
This month, I successfully completed the migration from Composer 1/Airflow 1 to Composer 2/Airflow 2 in five major projects. This challenging task, coupled with significant responsibilities, has been my favorite thus far.
📚 Resources to Start Learning Airflow
Below is a compilation of resources that can serve as a starting point for learning Airflow.
📜 Docs:
🗃️ Astronomer Registry, good for providers/modules/solutions:
🎓 Astronomer Academy:
📺 Astronomer Webinars/Airflow Monthly Town-Hall/Community Slack:
Engaging in calls and community discussions has introduced me to amazing people, fostering connections and learning opportunities.
💻 Udemy:
- M. Lamberti has great courses on Udemy https://www.udemy.com/user/lockgfg/
🔮 Medium (https://medium.com/):
- Medium serves as a rich repository of remarkable projects and solutions with Airflow/Composer. Therefore, I suggest looking for examples of DAGs and solutions on Medium as well.
✨ Cloud Composer:
- A fully managed workflow orchestration service built on Apache Airflow. https://cloud.google.com/composer#documentation
My preferred method of learning is through self-driven projects, where I can gain hands-on experience and grow from my mistakes. I carefully consider the idea, the tools I plan to utilize, and the data involved in each project. While this approach is time-consuming, especially during the testing phase to achieve my desired results, I believe it offers invaluable lessons and personal growth.
🚀 The Astronomer Champions Program for Apache Airflow
Last month, I was accepted to join the Astronomer Champions Program for Apache Airflow. Champions are leaders within the community who dedicate their time and expertise to increasing the understanding of and awareness for the Airflow product and mission.
Apply to be a Champion today! As a member, you’ll enjoy a range of exclusive benefits, including; comped certifications, exclusive access to Airflow contributors, beta access to new features, academy courses, and certifications…and of course, swag!
Please don’t hesitate to contact me on LinkedIn if you have any questions or would like to learn more about the program! 😊
