Everything I Hate About Machine Learning
I recently gave a talk about Machine Learning in Production, and had a really great time making this slide:
Throughout the years I’ve worked with ML, I’ve very quickly become an ML pessimist. Thought it’d be fun to nerf my morning today and spend some time expounding on some of the points above — from most legitimate to most absurd.
Treating ML in industry like its academia: Although I think the industry has gotten better at this, there are still many cases where we don’t consider the net value of ML for a company before embarking on a project. What’s the cost to scope, build, deploy, and maintain the solution? Is the value generated by the model greater than that? When we don’t answer these questions, we end up deploying the largest model we can find on Huggingface.co and meeting 0 project requirements. It doesn’t matter how accurate the model is if it isn’t fast enough, cheap enough, interpretable enough, or downright useful enough.
People who ask for ML because ML: ML is an implementation detail. You don’t ask for an implementation detail, you ask for a solution. ML might be that solution, but it very well may not be. If you’re starting with ML before you think critically about what solving the problem at hand looks like, you’ll probably fail. In many cases OOTB software engineering can solve problems faster, cheaper and easier.
The lack of SWE principles: I know data science is more experimental than SWE. I know it’s an iterable, scientific process with lots of dead ends and adhoc experiments. However, that doesn’t mean you can avoid writing unit tests, using source control, or automating processes with CI/CD for the stuff that sticks. When I was a data scientist/MLE, I’m pretty sure every software engineer I knew could’ve delivered value 5x faster if they were in my shoes due to their working knowledge of software tools, systems and practices. A model is worth nothing sitting in your local jupyter notebook.
Limited best practices for ML governance: There are so few best practices the industry is fully-aligned on that I had to make some — how crazy is that?
This image:
Not because it’s wrong but because that paper came out in 2015 and it’s still everywhere and we haven’t solved these problems. It’s painful.
Medium articles (yeah, mine too): Does anyone else feel like they have to get concensus-based definitions for ML topics on medium from like 5 separate articles? The other day I was refreshing my memory on cross-validation and it took me like twenty minutes to piece a definition together from every piece of content that had it partially right. Disagreeing on the definition, purpose and usage of fundamental techniques is a red flag for the field.
And on that note “flask + docker” tutorials: For the topics we do agree upon we’ve really beat a dead horse. We know how to serve predictions with a REST API. You don’t need to write that article.
Poorly-defined roles: Every job MLE job description requires a full read. I’m like “is this the MLE that does dashboards, builds models, deploys models, builds data pipelines, works with infrastructure, or is just a glorified PM?” Then you hear the salary (if they even posted it) and you’re like “haha no thanks anyways”.
The bad recruiters: Even if I were seeking, I wouldn’t be interested in leaving my FT job for a part-time, contracted data analyst position in rural NJ or to be a front-end developer (how??) at your web3 startup.
Buzzwords: FizzBuzzwords. Hah.
Andrew Ng’s soothing voice: At first it was comforting. Now it’s frustrating.
The 4 basic assumptions of linear regression: MLE’s shouldn’t need to know this. Everyone knows we’re overpaid statisticians who’re gonna jump right to the random forest model anyways. Stop saying homoscedasticity. It sounds dumb.
People who stan Pytorch: Yeah we get it you do research and like the APIs and are working towards a benchmark on some pre-canned dataset.
People who stan TensorFlow: Thought you were safe? No, we all know TensowFlow is still terrible.
People who refuse to touch Docker: It’s literally the easiest syntax I’ve ever learned and it provides so much benefit I promise it’ll take you like an hour (besides the whole networking thing networking can be hard).
Elon Musk: I don’t know what he’s doing but he’s involved and I don’t like it.
End of this rant but if you have more hit the comments or lets Twitter thread about it: https://twitter.com/kylegallatin.
