Setting up airflow on Synology NAS
This article is part of the series of using the data from the solar panels, meant to help me analyze my green score, plus the return on my investment. You can read about the journey here.
From mono to micro
Now that I have the data processing running in dbt, I have decided to split my already monolith into 2 services and to add another one to it:
- download the data from the website with the RPA
- data processing (EL with sql alchemy and T with dbt)
- process orchestration (with airflow)
Do I really need dbt to process my data? Not really, but I do dislike SQL in double quotes and I decided to keep it for data processing.
Why did I made the split? I am a huge fan of micro-services and while I am a one person only team, I still see the benefit of the split because I can easily separate and retry as much as I need the RPA download, while running the data processing only when the data is ready. Plus, it is easier to maintain code per single repository.
Why do I plan to use airflow?
Initially I was thinking of celery, since I’ve used it for quite some time (or luigi, or dramatiq) They all serve my purpose and one is harder to configure than the other. Nothing…
