CODEX
What should we learn from the short-lived career of Data Scientists?
I am arguing here that within a handful of years, Data Scientists will become just Software Engineers (SWEs). Just like Front-end and Back-end engineers, we will have Machine Learning Engineers (MLEs). However, in this post, I want to make sure that we will learn the right lessons from their short-lived career.
Before the two careers merge, they had different histories, different practices, and different cultures. So, let me first shed some light on those differences.
A Path-dependent Process
The term “Path dependence” is used in the history of economics. When there are two groups, each go through different historic events, and consequently take different actions, the two groups end up having two different economies. No matter how small and insignificant the different actions they took, these different paths lead to different destinations.
Today, I want to argue that software engineers and machine learning engineers also went through path dependent processes and ended up having different cultures.
By the end of the 20’s century, the software industry became mature enough, and the need for the product manager role was created. The product managers understand the business and the customer needs. They usually write the customer requirements in the form of user stories and hand them to the developers.
This led to two consequences. It widened the gap between the software engineers and the business needs. But, since the engineers have the exact requirements, they were able to focus on the engineering part of their job. Think of the engineers building a bridge, they have no room for experimentations, they rather follow the best practices. Similarly, software engineers mastered code reviews, unit tests, and DevOps.
Machine learning engineers have a different history. Their history is intertwined with data analysts. These are the one responding to business questions all the time. They are also not the best software developers. Therefore, they pay more attention to the business outcome, and less attention to the quality of their code, its maintenance or scalability. They like MVPs, and experimentations. They were formerly known as data scientists not engineers for a reason. However, their experimentations can get them stuck into local minima sometimes.
You can see how the two cultures differ in how they build their code, maintain it, and deploy it. Even when it comes to data collection; Software engineers prefer lean code. They only store the data they need at this very moment. Machine learning engineers know that data is worth more than the algorithms they use, thus they want to collect not only the data they need now, but the one they might need 18 month in the future.
The use of the word testing among the two cultures, summarizes it all. For software engineers, their tests are meant to validate that their code does what is written in the user stories. They are like the musicians in a classical western orchestra. The best musicians are the ones playing the same notes Beethoven had in mind when he wrote them down.
In eastern music, or even in Jazz, the performers are the composers. They may involve the audience in their music, and change it according to their feedback. Machine learning engineers are the same, their tests are meant to capture the effect of their algorithm on the users, and they improve it accordingly. In some extreme cases, they are okay with deploying buggy code if by a stroke of luck it was able to a desired business outcome.
“Putting on your big-boy pants”, Noel Kippers
But we all know that those wild data scientists are becoming engineers now. Or as my friend, Noel Kippers, likes to say, it is time for them to put their big-boy pants on. I am very positive that within 5 years there will be no such thing as machine learning engineers, they will become just software engineers who happen to use libraries such as TensorFlow, HuggingFace and Scikit-Learn.
Just like in git, machine learning engineers are a feature branch that will be merged into the main branch soon.
But also like in git, when merging a branch with the main branch, we incorporate the code from the feature branch into the main one. We have seen how data scientists adopted the build-measure-learn spirit of lean startups. They worked in iterations. They are never certain if a model will work, that’s why to always built prototypes and made sure to fail fast. Their metrics were usually related to the business metrics.
Ideally, software engineers should be much different, but the sad truth is that in most of the companies they are. In the majority of the companies out there, they care about output not outcome, their sprints are meant to produce as many features as possible, not as much business outcome. They are rarely part of the product discovery process.
“The little secret in product is that engineers are typically the best single source of innovation; yet, they are not even invited to the party in this process.”, Marty Cagan (Inspired)
That’s why I am worried that when this merge happens in the future, most of businesses will learn nothing from the short-lived career of data scientists, and will keep writing their code as a waterfall with an agile lipstick on its face.
I do not have a silver bullet yet for how to prevent this from happening. I also know that in most of the companies data scientists did not offer the value they promised. So, each culture have things to learn from the other, and based on your feedback I might write follow-up posts to this one.
More from the same author:
If you find this article useful, then you may want to check my book, Hands-On Machine Learning with Scikit-Learn and Scientific Python Toolkits.
