Extract. Transform. Read.
Extract. Transform. Sketch?
Why data engineers need to get ideas on paper and how to present them.
The following short read is the latest edition of my weekly newsletter, Extract. Transform. Read. sent to 2,000+ aspiring data professionals. If you enjoy this snippet, you can sign up and receive your free project ideation guide.
I’ve recently been honing a data engineering skill that might not occur to you — drawing.
When I first started my data engineering job 3+ years ago, any description or information related to my code would be in written form. This meant everything from README documentation to illegible legal pad scribbles would be all I had to inform decisions about design and implementation.
Lately, however, my tasks have grown in both complexity and volume. What I need to convey to myself and my team won’t fit on one sheet of paper. And if it did, it wouldn’t make a bit of sense.
So I’ve turned to diagramming tools.
I use tools like Microsoft Visio and Draw IO to create clear depictions of pipelines.
This makes it easy to:
- Communicate an idea universally — I work with colleagues around the world so sometimes it’s helpful to communicate in a way that isn’t exclusively English
- Highlight pain points — Applying a different color draws the eye more than an all-caps TO-DO or FIX ME would
- Make quick edits/rearrange components — have you ever tried rearranging paragraphs or pages of text minutes before a meeting? There’s an entire industry dedicated to this called copyediting — and it can’t be done 5 min before a Zoom call
- Visualize independent/dependent tasks — this is critical when revising data pipelines that rely on upstream checks to trigger certain associated actions
And, finally, the act of creating an architecture (arch) diagram subtly communicates something: That significant thought went into what you want to propose or present.
Anyone can scribble in a shared doc or reference a notebook of ideas. Taking the time to spend time compiling a visualization demonstrates intent and care.
This translates to you communicating your investment in making sure your build is possible and functional.
Otherwise, it’s back to the (literal) drawing board.
Thanks for ingesting,
-Zach Quinn
