Data Meshes, Data Testing, TDD for Data Teams; ThDPTh #8

The 101 of Data Meshes, testing data with auto profiling, developing data pipeline in a test-driven way.

Created by the author with pitch.com. The image has nothing to do with the newsletter, it just came to my mind while observing the late covid-19 statistics in Germany, and it’s a data point.

Data will power every piece of our existence in the near future. I collect “Data Points” to help understand this future.

If you want to support this, please share it on Twitter, LinkedIn, or Facebook.

This week I stumbled over a couple of important topics I mentioned before, data meshes, test-driven development for data teams, and testing data with great-expectations.

1 TDD for Data People

I was about to write an article myself about a test-driven workflow for data people’s “daily bread”. But low and behold, Marcos Marx was faster. I already mentioned in an earlier newsletter, that I believe test-driven development to be the fastest and most productive way for any engineer, be it data, infrastructure, or software engineer to work, period.

This article walks you through the creation of an Apache Airflow DAG in a test-driven fashion, check it out. It’s really good work!

Ressources

• The medium article by Marco Marx.
• The step-by-step tutorial on github.