Extract. Transform. Read.
Data Engineering’s 3rd “I”
Explaining state: How to maintain it, why pipeline runs need to be identical and breaking down an abstract term.
The following short read is an excerpt from my weekly newsletter, Extract. Transform. Read. sent to 2,000+ aspiring data professionals. If you enjoy this snippet, you can sign up and receive your free project ideation guide.
One of the dirty secrets about my job is how easy it can be to fix broken pipelines. Often I’m re-triggering a failed DAG task or, if using a codeless pipeline, literally hitting refresh.
In fact, “refresh” is a great example for one of the more abstract data engineering concepts: State.
And, specifically the maintenance of state under any condition. This is the definition of an important I-word, “idempotency.”
While idempotency sounds like an SAT word, it’s as simple as saying “Every time this process runs the result (end state) will be the same.”
An easy-to-grasp example of idempotency is the Google Cloud BigQuery API’s “WRITE_TRUNCATE” property. If you run a pipeline with “WRITE_TRUNCATE”, your data will always be overwritten during the load step.
A more precise version of implementing idempotency is something I include in nearly all my pipelines, a DELETE step. This is slightly more precise than overwriting data because I am specifying deletion for a particular window.
But this means that when I run a job that deletes and inserts only yesterday’s data, the output will be the same each time, leaving historic data intact and avoiding the very real possibility of data loss.
This is a very practical approach to designing data pipelines because you may get spur-of-the-moment requests to reload data or otherwise re-trigger your runs.
When executed properly, idempotency is as easy as hitting page refresh.
Thanks for ingesting,
-Zach Quinn
