Member-only story
Implement Behaviour Driven Development in data pipelines using Mage
Maximize the quality and productivity of your data pipelines
In my previous articles, I talked a lot about the importance of testing in data pipelines and how to create data tests and unit tests respectively. While testing plays an essential role, it may not always be the most exciting part of the development cycle. As a result, many modern data stacks have introduced frameworks or plugins to expedite the implementation of data tests. In addition, unit testing frameworks in Python such as Pytest, unittest have been there for a long time, helping engineers efficiently create unit tests for data pipelines and any Python applications.
In this article, I want to introduce a setup that uses two modern techniques: Behaviour Driven Development (BDD) — a business-oriented testing framework, and Mage — a modern data pipeline tool. By combining these two techniques, the objective is to create high-quality unit tests for data pipelines while having a seamless developer experience.
What is Behaviour Driven Development (BDD)?
When building data pipelines for business, it’s highly likely that we will encounter complicated and tricky business logic. One example is to define customer segmentation based…
