Marin AglićUsing Celery RedBeat for basic dynamic scheduling of periodic tasksWhen using Celery without Django, a question that kept bugging me was — how to disable and re-enable periodic tasks? In this story we take…Feb 15Feb 15
Marin AglićLearning Apache Iceberg — looking at append, update and delete operationsBridging the gap between what I knew and what I wanted to learn. This is the fourth in a series of articles about Apache Iceberg. In the…Feb 5Feb 5
Marin AglićLearning Apache Iceberg — storing the data to Minio S3Bridging the gap between what I knew and what I wanted to learn. This is the third in a series of articles where I continue my progress in…Jan 30Jan 30
Marin AglićinPython in Plain EnglishGetting DAG data from the Airflow API — working with PolarsThe Airflow API allows us to programmatically gather information about our Airflow deployment, such as DAG runs, task instances, pools…Nov 6, 2023Nov 6, 2023
Marin AglićProblem with using data intervals when backfilling dataset scheduled DAGsData aware scheduling is an Airflow feature introduced in Airflow 2.4. The feature basically introduced event-based scheduling between…Oct 15, 20231Oct 15, 20231
Marin AglićinData Engineer ThingsDesigning Dynamic Workflows with Celery and PythonThe why and how to insert chains into chainsAug 7, 2023Aug 7, 2023
Marin AglićConsuming and publishing tasks in Celery at a steady rateFor a time now I’ve been thinking on how to implement rate limiting in Celery. Finally, I found the time and resources that helped me. In…Jun 5, 2023Jun 5, 2023
Marin AglićinBetter ProgrammingLearning Apache Iceberg — Storing the Catalog to PostgresAn interesting experiment to see how Apache Iceberg organised the table data and metadata under the hoodMay 17, 20231May 17, 20231
Marin AglićinBetter ProgrammingMaking Async API Calls With Airflow Dynamic Task MappingApr 18, 2023Apr 18, 2023
Marin AglićDynamic task mapping over a custom deferrable sensor (in a task group)This story is a result of an inconsistency I found while using dynamic task mapping over a task group containing a custom deferrable…Apr 16, 2023Apr 16, 2023
