Path Dependent History


What should we learn from the short-lived career of Data Scientists?

Tarek Amr
Tarek Amr
Mar 3 · 5 min read

I am arguing here that within a handful of years, Data Scientists will become just Software Engineers (SWEs). Just like Front-end and Back-end engineers, we will have Machine Learning Engineers (MLEs). However, in this post, I want to make sure that we will learn the right lessons from their short-lived career.

Before the two careers merge, they had different histories, different practices, and different cultures. So, let me first shed some light on those differences.

A Path-dependent Process

The term “Path dependence” is used in the history of economics. When there are two groups, each go through different historic events, and consequently take different actions, the two groups end up having two different economies. No matter how small and insignificant the different actions they took, these different paths lead to different destinations.

Git Branch

Today, I want to argue that software engineers and machine learning engineers also went through path dependent processes and ended up having different cultures.

By the end of the 20’s century, the software industry became mature enough, and the need for the product manager role was created. The product managers understand the business and the customer needs. They usually write the customer requirements in the form of user stories and hand them to the developers.

This led to two consequences. It widened the gap between the software engineers and the business needs. But, since the engineers have the exact requirements, they were able to focus on the engineering part of their job. Think of the engineers building a bridge, they have no room for experimentations, they rather follow the best practices. Similarly, software engineers mastered code reviews, unit tests, and DevOps.

SWE & MLE; 2 different cultures

Machine learning engineers have a different history. Their history is intertwined with data analysts. These are the one responding to business questions all the time. They are also not the best software developers. Therefore, they pay more attention to the business outcome, and less attention to the quality of their code, its maintenance or scalability. They like MVPs, and experimentations. They were formerly known as data scientists not engineers for a reason. However, their experimentations can get them stuck into local minima sometimes.

You can see how the two cultures differ in how they build their code, maintain it, and deploy it. Even when it comes to data collection; Software engineers prefer lean code. They only store the data they need at this very moment. Machine learning engineers know that data is worth more than the algorithms they use, thus they want to collect not only the data they need now, but the one they might need 18 month in the future.

What does testing means to SWE vs MLE?

The use of the word testing among the two cultures, summarizes it all. For software engineers, their tests are meant to validate that their code does what is written in the user stories. They are like the musicians in a classical western orchestra. The best musicians are the ones playing the same notes Beethoven had in mind when he wrote them down.

In eastern music, or even in Jazz, the performers are the composers. They may involve the audience in their music, and change it according to their feedback. Machine learning engineers are the same, their tests are meant to capture the effect of their algorithm on the users, and they improve it accordingly. In some extreme cases, they are okay with deploying buggy code if by a stroke of luck it was able to a desired business outcome.

“Putting on your big-boy pants”, Noel Kippers

But we all know that those wild data scientists are becoming engineers now. Or as my friend, Noel Kippers, likes to say, it is time for them to put their big-boy pants on. I am very positive that within 5 years there will be no such thing as machine learning engineers, they will become just software engineers who happen to use libraries such as TensorFlow, HuggingFace and Scikit-Learn.

Git Merge

Just like in git, machine learning engineers are a feature branch that will be merged into the main branch soon.

But also like in git, when merging a branch with the main branch, we incorporate the code from the feature branch into the main one. We have seen how data scientists adopted the build-measure-learn spirit of lean startups. They worked in iterations. They are never certain if a model will work, that’s why to always built prototypes and made sure to fail fast. Their metrics were usually related to the business metrics.

Ideally, software engineers should be much different, but the sad truth is that in most of the companies they are. In the majority of the companies out there, they care about output not outcome, their sprints are meant to produce as many features as possible, not as much business outcome. They are rarely part of the product discovery process.

“The little secret in product is that engineers are typically the best single source of innovation; yet, they are not even invited to the party in this process.”, Marty Cagan (Inspired)

That’s why I am worried that when this merge happens in the future, most of businesses will learn nothing from the short-lived career of data scientists, and will keep writing their code as a waterfall with an agile lipstick on its face.

I do not have a silver bullet yet for how to prevent this from happening. I also know that in most of the companies data scientists did not offer the value they promised. So, each culture have things to learn from the other, and based on your feedback I might write follow-up posts to this one.


Everything connected with Code & Tech!

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store