Extract. Transform. Read

Testing… Testing… Environments

Categorizing data engineering IDEs to facilitate professional, production-adjacent development.

Zach Quinn
Pipeline: Your Data Engineering Resource

--

The following short read is an excerpt from my weekly newsletter, Extract. Transform. Read. sent to 2,000+ aspiring data professionals. If you enjoy this snippet, you can sign up and receive your free project ideation guide.

It’s never good when you wake up to this from a coworker:

💀

The skull wasn’t because the sender felt like they would suffer any kind of dramatic fate. Instead, they were prepared to administer near-fatal justice to the junior engineer who made several unnecessary overnight commits straight to our org’s main branch.

The thing is, for a first-time violation, I can understand why testing is an afterthought for new engineers. Schools and courses emphasize local output over production so testing feels like an extra step.

To properly test code, you need to configure a clean, production-adjacent environment. If you’re new to this concept, here are 2 of my favorites along with an unusual choice.

The safe choice: Virtual Environment

I use two virtual environments that can be configured interchangeably: Pyenv and Venv.

Pyenv is easy to configure and use within a terminal in a “professional” IDE like VS code. Pyenv is ideal because it allows me to create an environment from a blank slate each time.

Venv is another option. Instead of using Venv in VS Code this is how I set up a virtual Python environment inside of a Virtual Machine (VM).

Read more to learn how to set up a quick, durable sandbox in a Compute Engine VM.

A hand filling in bubbles on a scantron test sheet.
Photo by Nguyen Dang Hoang Nhu on Unsplash

The portable option: Docker

I’ll confess: I didn’t used to be a fan of Docker. I didn’t really “get” containerization and could set up a virtual environment using the processes described above.

However, I learned that Docker’s true power is its portability. Not only can I create a clean slate (an image), I can push this to a registry to create testing configs before I test script changes in production.

Powerful stuff.

The one issue I had was authenticating with GCP; I describe my solution here.

The unusual pick: Jupyter Notebook

Jupyter Notebook gets a bad rap in the data engineering community. Seen as a tool for data analysts and data scientists, it doesn’t quite make sense for data engineers to develop and test in an environment best known for its nicely rendered outputs.

But buying into that argument would cause you to miss out on some useful features and, frankly, a nice UX.
I recently discussed my internal struggle between VS Code and Jupyter so you can do your own soul-searching.

Thanks for ingesting,

-Zach

--

--

No responses yet