Great writing. It’s funny some knowledgeable problems like financial reporting become a nightmare at some scale. It’s nice to see your usage of spark with this. Some of my client usually go into another route, like using columnar storage in the cloud like Redshift or BigQuery. Or Vertica and Teradata. At the end the day having developers with good architecture vision and own this pipeline does all the difference, at a lower cost too.