Hey Samson! I read this post when you wrote and and just happened across it again, a year+ later. I’m sure you still doing similar work, so I just want to mention a new option for the data modeling component of your stack: dbt. dbt (data build tool) is an open source project that I and several other folks started building while at RJMetrics and have since gone out on our own to continue building. It’s like Luigi for SQL—just far easier to set up, extend, and manage. What we’ve found is that this actually covers a large majority of the need, and packaging it up in this much simpler fashion makes it accessible for analysts who spend their days writing SQL.
Would love if you’d check it out and curious to hear any thoughts you have.
I originally wrote about the thinking behind dbt here.