Jun Wei NgSimilarities between an Agile retro and parentingHaving 2 children of my own, I can speak from experience that raising a child is not too bad, but raising 2 together is exponentially…4 min read·Jan 11, 2023----
Jun Wei NgUsing nested operators to achieve a dynamic Airflow DAGNesting operators might do the trick, but at what costs?5 min read·Mar 14, 2022--1--1
Jun Wei NgCreate a dynamic Airflow DAG using a YAML file (or any other flat file)We can control a DAG’s workflow using a flat file that contains the dynamic variable. Sounds easy, but it does have its downside too4 min read·Mar 14, 2022--4--4
Jun Wei NginTowards DevUsing a Python file to store dynamic configuration for an Airflow DAGEasy to make dynamic changes, but with its own drawbacks5 min read·Mar 14, 2022----
Jun Wei NgManage a dynamic Airflow DAG using a MongoDB (or any other database)By storing the dynamic configuration in an external database, we can easily change it. But we also incur the overhead of additional load…5 min read·Mar 14, 2022----
Jun Wei NgUsing environment variables to control a dynamic Airflow DAGNot the best way to achieve dynamic task creation, but maybe as a good way to toggle DAG behaviour based on environment…4 min read·Mar 14, 2022--1--1
Jun Wei NgUsing Airflow variables to create a dynamic DAGEasy to configure for dynamism, easy to debug, but with some drawbacks4 min read·Mar 14, 2022--1--1
Jun Wei Ng6 ways to write a dynamic Airflow DAGWeigh out the pros and cons before choosing one to create your dynamic Airflow DAG3 min read·Mar 13, 2022----
Jun Wei NginSelectFromWriting unit tests for Airflow custom operators and hooksHow to mock Airflow connections and variables, DAG run context, and Airflow hook initialisation during unit testing4 min read·Mar 8, 2022--1--1
Jun Wei NgWriting unit tests for an Airflow DAGUnit tests are the fundamental of the test pyramid. They should be fast to execute and cover as much of the code as logically sensible. The…3 min read·Mar 7, 2022--2--2