Write Code like a Sr. Data Engineer: Design Patterns for Managing Complexity

Pushing Complexity Upwards or Pulling it Downwards? How to make your code simpler (and better)

Congratulations! 🎉

If you’re reading this, you likely want to get better at writing code for data engineering.

Background:

The following principles are discussed in A Philosophy of Software Design, by John Ousterhout.

Complexity: Makes a system hard to understand or modify.

“In a complex system, it takes a lot of work to implement even small improvements.” (page 6)

TLDR: We don’t want our code to be complex.

Problem Statement:

You have an API endpoint which receives an encrypted ID as a query parameter. Your code must decrypt the value and determine if it is valid.

• An error must be raised if the value cannot be decrypted (implies usually that invalid ID was provided).
• An error must also be raised if the value is “invalid” (oversimplified, an empty string here).

There are 2 areas of…