Dawn of DataOps: Can We Build a 100% Serverless ETL Following CI/CD Principles?

Is it time to enjoy the benefits of DevOps in the informational space?

I admit it, as a data engineer I have been full of envy and jealousy towards my colleagues — application engineers — in countless times.
When they were automating their production release and deployment generation processes, we had to waste hours drawing arrows and copying boxes from one ETL environment to another or manually resolving conflicts between different branches (if code versions on my colleagues’ laptops can be even called that) of a semantic model of a BI tool.

And is that any ETL or BI tool that comes to mind still has that 90s flavor — Why this obsession to invest in a development GUI, which is end up using by programmers and not by business users? Why these little boxes connected with arrows with predefined data transformations that are never used? Why do we always end up building a component that ends up calling an external mega script (osCommand) filled with spaghetti code? Is SQL not worthy enough to have proper version control?

Anyway, many questions, but few answers … until now

The data revolution that we have experienced in recent years i is undeniable. In virtually any…