Data orienting, a data engineering pivot

In programming, DOP (Data-Oriented Programming) is a paradigm that helps us simplify our system. Improving the flexibility and velocity of our coding and perhaps even better understanding of our means and target.

Alas, I’m not here to talk about programming.

I will talk about myself, and how I came to implement this paradigm in my life as a data engineer. My failures, misconceptions, commitments issues, communication challenges, and the struggle to see a process through (honestly, I’m talking about data). I worked in gigantic companies (6k+), small startups (30), and as a consultant (50~ different companies, 10~ different business types, and a lot of data!). I will share with you glimpses of my data journey along with tips to encourage your surroundings to be more data-oriented themselves.

1. Consider what you want to bring to the table

(where do you want to make the impact)

Trigger alert, IMHO a data engineer needs to understand the business. You can’t expect to make a truthful impact in your company without deeply understanding how your actions are relating to the decision-makers.

While I was working as a consultant there was a major learning curve, not a technical but a methodological one. Companies I worked with came with an absolute truth I had to bend to and fulfill. And I did. Soon I came to learn some of them weren’t pleased with the results, my solo pipeline design did the trick but nobody else knew (nor tried) to maintain it. I did the job efficiently but never heard from some of those companies again.

I didn’t fail to deliver, I failed to understand the needs, scope, and intent of the business as well as the capabilities of my environment. Very fast I’ve learned the fine art of simplification. If an ETL works well and it is highly complex, does it really work well?

Simplifying my ideas helped me communicate them better, allowing me to take a meaningful place in the design and planning phase and eventually create a better impact as a data engineer. Not to mention the abstraction did help in the hands-on department. I became whole as a consultant (kidding it was a living hell).

2. Consider the table

(the scope of the impact you aspire to make)

I’ve always seen myself as a multi-disciplined persona, I understand maths and statistics, I’m very good around the code lines and my analytical skills have brought me so far. Keeping things in focus was a tough one. I have a lot of opinions about a lot of subjects. While working at a startup company (a fintech if you must know) I was highly encouraged to innovate and be creative. But innovate what? I have so many ideas, paths to go after, and life-changing questions to ponder (ok, it wasn’t that dramatic). I remember the CTO asked me to make an integration of a Jira Sprint to a react web app. There was data somewhere in that equation I assure you, but it wasn’t what I wanted to do. It was missing an impact and a deeper meaning to what the company did. I ended up voyaging a lot of technologies and it was a pleasant time of no pressure and endless researching. But a piece was missing.

You probably know by now, that you can’t always choose the most fascinating projects, but over time I’ve noticed, that there isn't an absolute force preventing you from picking you're brain now and then. Determining the subjects I would like to focus on and improve myself helped me to better understand my career path (and my managers to tunnel me better).

3. Consider the screws

(the tools you’ll acquire)

In our line of techie business, there are so many tools to consider in so many different aspects and steps. Forget about those kinds of tools, it is an endless sprint of hypes with several Usain Bolts at the top. My first steps in the data domain were as a DBA team leader in the IAF. My skills and solo projects brought a lot of focus to me and I was asked to build a team to work around databases in a vast ecosystem of needs and dependencies (a.k.a the army).

Well, that was terrific, the problem was ... they weren’t so sure they needed this team in the first place. There were a lot of overlapping teams, and a proficient dedicated data team wasn’t so obvious. So I had to make it obvious. It was difficult, we had a lot of overnight projects and technical blockers. We were the underdogs and it felt like a Tarantino movie. Eventually, we moved past this temporary position and established the right working connections and methodologies around our team.

My biggest lesson here was that the best tools you will earn along the way are the ones that help you emphasize your work, harness people to your decisions, designs, ideas, and eventually the ones that put the best light on the impact you are trying to make.