Jun Wei NgSimilarities between an Agile retro and parentingHaving 2 children of my own, I can speak from experience that raising a child is not too bad, but raising 2 together is exponentially…Jan 11, 2023Jan 11, 2023
Jun Wei NgUsing nested operators to achieve a dynamic Airflow DAGNesting operators might do the trick, but at what costs?Mar 14, 2022Mar 14, 2022
Jun Wei NgCreate a dynamic Airflow DAG using a YAML file (or any other flat file)We can control a DAG’s workflow using a flat file that contains the dynamic variable. Sounds easy, but it does have its downside tooMar 14, 20223Mar 14, 20223
Jun Wei NginTowards DevUsing a Python file to store dynamic configuration for an Airflow DAGEasy to make dynamic changes, but with its own drawbacksMar 14, 2022Mar 14, 2022
Jun Wei NgManage a dynamic Airflow DAG using a MongoDB (or any other database)By storing the dynamic configuration in an external database, we can easily change it. But we also incur the overhead of additional load…Mar 14, 2022Mar 14, 2022
Jun Wei NgUsing environment variables to control a dynamic Airflow DAGNot the best way to achieve dynamic task creation, but maybe as a good way to toggle DAG behaviour based on environment…Mar 14, 20221Mar 14, 20221
Jun Wei NgUsing Airflow variables to create a dynamic DAGEasy to configure for dynamism, easy to debug, but with some drawbacksMar 14, 20221Mar 14, 20221
Jun Wei Ng6 ways to write a dynamic Airflow DAGWeigh out the pros and cons before choosing one to create your dynamic Airflow DAGMar 13, 2022Mar 13, 2022
Jun Wei NginSelectFromWriting unit tests for Airflow custom operators and hooksHow to mock Airflow connections and variables, DAG run context, and Airflow hook initialisation during unit testingMar 8, 20221Mar 8, 20221
Jun Wei NgWriting unit tests for an Airflow DAGUnit tests are the fundamental of the test pyramid. They should be fast to execute and cover as much of the code as logically sensible. The…Mar 7, 20222Mar 7, 20222