Why ML Ops Matters: Taking the ‘Oh Sh*t’ out of MLOops — “What is MLOps?” — Part 3
An introduction to the “What” and “Why” of MLOps.
Hey everyone, in this series I’ll be diving into the “What” and “Why” of MLOps.
This is Part 3 of the multi-part series that will include:
- Part 1: Introducing the series: “What is MLOps?”
- Part 2: Defining MLOps as Simply As Possible
- Part 4: Goals of MLOps as Themes (Substack)
- Part 5: Software and ML Before MLOps (Substack)
- Part 6: The Challenges of Scaling ML As Software (Substack)
Links will be updated as the sections are published. Sections tagged with “Subtack” will be available in their entirety at my substack for free.
Why should individuals or teams care about MLOps? Why invest in implementing MLOps in your team, company, or projects?
1. Mitigating Risk and Bias
For many companies (and industries) deploying machine learning at scale is such a new endeavor that those organizations may not be aware that tooling won’t solve all their problems (both existing and upcoming).
MLOps is about practices just as much as it’s about tooling.
A data and machine learning mature organization is one that deeply appreciates Murphy’s Law and the power of monitoring and rollbacks.
Machine learning models can’t just be trained offline, deployed, and then ignored to the detriment of a company’s brand and reputation.
As more than a few people have noted:
“People think computers will keep them from making mistakes. They’re wrong. With computers you make mistakes faster.” — Adam Osbourne
“Computers make very fast, very accurate mistakes.” — Roy Zuvers
And widespread adoption and accessibility to data storage and computing abstractions have made it easier to bootstrap ML products while side-stepping concerns of governance and bias.
We’re getting so much faster at manifesting ideas that we don’t stop to think about whether those ideas are valuable or morally reprehensible.
All software has inherent risk, including:
- Services being unavailable for a given period of time;
- Services and APIs returning unexpected or bad results for a given call;
- Service performance degrading as volume increases over time;
- Tribal knowledge and skills necessary to maintain the services being lost or poorly transferred.
In recent times however, ML applications seem to feature in some spectacularly terrible stories.
Some famous examples of where having a robust MLOps (and data governance) practice could have saved even data-driven companies and teams a bit of heartache:
- Microsoft’s famous super racist Twitter bot incident
- Healthcare algorithm failed to flag Black patients
- Zillow wrote down millions of dollars, slashed workforce due to algorithmic home-buying disaster
- Misidentifying Athletes as Criminals
- In child welfare, a New Zealand experiment in predictive analytics touted as a great success wrongly predicted child abuse more than half the time.
- In Los Angeles County, another experiment was hailed as a huge success in spite of a “false positive” rate of more than 95 percent. And that experiment was conducted by the private, for-profit software company that wanted to sell its wares to the county.
One important point to note about the examples above: They are all examples of an ML application not working as intended.
If you’re looking for examples of ML being intentionally used in a biased, harmful and exploitative manner, check out the Awful AI repo.
In future posts we’ll cover how to make ML applications work as intended and feel less bad and less like failures.
2. Alleviating Market Shortages of Talent and Bolstering Existing Talent
Trying to find the “right” candidate profile is a hot topic of conversation at happy hours with startup founders, recruiters, and enterprise engineering hiring managers.
As someone heavily involved with hiring MLOps and data engineers through Mailchimp’s global eng hiring committee, the lift and investment in getting a pipeline of candidates has been heavy and time consuming (although worthwhile).
Generally speaking, candidates effectively fell into one of four buckets:
- Software engineers that wanted to move more towards applied ML and were trying to use the MLOps team as a jumping point into the data science organization (but didn’t have any prior experience in ML or familiarity with the ML lifecycle);
- Software engineers that were more junior and were genuinely interested in MLOps (but didn’t have any prior experience in ML or familiarity with the ML lifecycle);
- Data scientists that wanted to move up the value chain into engineering (but didn’t necessarily have a strong software engineering skill set) and didn’t have significant experience with deployment (either because the model was tossed to the engineers or much of the deployment infrastructure had been abstracted away);
- 🦄 Unicorn candidates 🦄 that had excellent software engineering skill sets as well as experience working with data scientists on machine learning projects (or had been data scientists themselves at some point). This was a much, much smaller bucket and the bucket that we ideally wanted.
Of the 4th bucket, almost all those candidates walked from the offer because they (rightfully) understood the high-value of what they’re unique positioning was and we were unable to meet their desired salary.
We ended up primarily hiring from the 2nd bucket when we exhausted the 4th bucket.
Why is it so hard to hire for an MLOps like role? Why this bottleneck?
In large part the skill set needed to be an effective engineer and ML practitioner requires balancing a very broad skill set with depth in core areas like distributed systems, cloud development, ML lifecycle awareness, data processing, language literacy, DevOps, etc.
And for candidates that do have skill sets in multiple areas, there was significant time, energy, and resources invested that they want to recoup as much as possible.
Hypothetically this would also mean that in order for candidates to have the time to develop these skills, they would need to be more mature candidates (i.e. not entry-level or junior candidates).
Another potential reason why we don’t see more of these unicorn candidates is incentive.
In many companies data scientists are rewarded for innovating and ideating potential projects and features for the company.
For engineers, their contribution to the codebase and ability to execute on someone else’s vision is rewarded.
If the data scientist’s performance isn’t dependent on the successful deployment of a model to production and an engineer isn’t incentivized to learn the ML process, it’s not going to happen.
Well-designed and unified abstractions coupled with mature practices however can help offset the impact of the shortage of unicorn candidates by empowering candidates and employees that don’t have “the ideal skillset”.
MLOps tooling can help growing and developing pactitioners provide value through the automation of tasks in the productionization and deployment of ML products, as we’ll describe in later parts of this series.
3. Unlocking Competitive Innovation Through Reducing Toil
The two sections above focus squarely on the impact to the company and how to mitigate shortages in magical-engineer unicorn labor.
In my section of hot-takes in the first part of this series I wrote:
“User experience of internal tooling or platforms is a critically undervalued and under-discussed component of adopting MLOps practices and tooling and consistently pointing the finger at Data Scientists not being able to code is lazy and uninspired thinking. “
The reality is that overworked, exhausted, and burned out engineers and data scientists are knowledge workers that only have a fraction of their superpower available at any point in time.
Their superpower?
Creativity, the real revenue generating skill for which they were were hired.
Burn-out isn’t always due to sheer overwork. Burn-out can also occur when work becomes repetitive, when knowledge workers feel as if they lack autonomy, and when it feels like their work has no meaning and isn’t pushing the needle forward.
If every model deployment feels like a Sisyphean task akin to rolling a giant rock up a mountain, data scientists will either shy away from the model deployment or will undertake the process and eventually quit or burnout when
- ⅓ of their projects get yanked because of shifting product concerns,
- ⅓ end up taking months to deploy, and the other
- ⅓ end up getting abandoned mid-way because the project was infeasible.
In some cases data scientists may decide to go find another solution outside the scope of the model deployment process and either:
1) Get so motivated by the problem they become MLOps engineers-in-waiting;
2) Stand up a rogue or shadow process that attempts to bypass any golden path the engineering team has set down.
A culture of practicing and advocating great MLOps practices and tools can help teams thrive instead of just trying to survive through minimizing the grind and unlocking new capabilities.
Data science and ML eng teams that are able to rapidly iterate and test new ideas are teams that will ultimately increase the bottom-line because bad ideas will be rapidly discarded and great ideas will have more buy-in for engineering and product investment.
A helpful thought experiment I enjoyed is Todd Underwood’s “How do we go from a junior engineer working on an application with an idea and some data and within a week or two has a functional experimental model that can do or not do something in a production ready environment so that iteration can happen?”.
Series List
This is Part 3 of the multi-part series that will include:
- Part 1: Introducing the series: “What is MLOps?”
- Part 2: Defining MLOps as Simply As Possible
- Part 4: Goals of MLOps as Themes (Substack)
- Part 5: Software and ML Before MLOps (Substack)
- Part 6: The Challenges of Scaling ML As Software (Substack)
Links will be updated as the sections are published. Sections tagged with “Subtack” will be available in their entirety at my substack for free.
About Me
My name is Mikiko and I currently work as a Sr MLOps Engineer at Mailchimp.
Before pivotting to MLOps I’ve also worked as a Data Scientist, Data Analyst, and Growth Hacker at various companies in the Bay Area.
If you’re interested in my various career leaps and breakthroughs, check out these series:
- 👩🏻💻 Miki’s 🔥Hot-Takes🔥 on MLE Interviews: Types of Roles & Interview Prep — Part 1 & Part 2 — Where I talk about getting my role (which was initial ML Engineering before the team converted to MLOps)
- ✂️Breaking into Data Science- From Hair Salon to Data Scientist 🔍 — Part 1, Part 2, Part 3, Part 4 — Where I go into exhaustive detail about preparing for getting a job as a data scientist.
If you’re interested in learning more about MLOps, production ML, and distributed systems + cloud development, I also publish:
- Videos on Youtube at The MLOps Engineer
- Technical walkthroughs on Substack
- Snark and saltiness on Twitter
- Professional sounding/career oriented stuff on LinkedIn
- Code gists and TODOs on Github
And I take support and patronage in the form of coffees ☕!
