MLOps seen by a CIO
This article was co-written with Ahmed Fessi.
At the start of this summer, Maria started posting on LinkedIn, at the end of the summer I (Raphaël) started doing the same, and more recently Başak joined in as well. In the process of sharing our knowledge (and jokes ;) we have made interesting connections and met some new professional friends.
One of such connections is Ahmed Fessi, Chief Transformation and Information Officer at Medius, a FinTech company specializing in spend management. Ahmed has led tech teams with >100 people and has led multiple tech transformations in corporate environments. Over the past few months, I’ve had the pleasure of exchanging multiple chats and calls with Ahmed, and I’m even reading his book.
In our exchanges, we discussed the challenges of working in tech leadership and the special challenges that come with doing tech in tech companies, as well as doing tech in non-tech companies (there are significant differences). We also looked at MLOps implementation and MLOps transformation seen from the perspective of the executive level. Today we would like to share some of the things we discussed, and some of the insights we gained.
Raphaël: You have been working in data and tech for 15 years, what are your takeaways so far?
Ahmed: In my journey as a CIO, and previously as Head of Data Platforms, I’ve closely witnessed the evolution of digital transformation and its profound impact on enterprises.
I like to share that when I got my computer science engineering degree (yes, that was many years ago), Cloud Computing, Big Data, or Generative AI, weren’t part of the curriculum, yet I (we) interact with these technologies daily nowadays. Over the years, many concepts, tools, and buzzwords appeared (and some disappeared), and somehow shook the tech landscape. One of the key skills I had to develop was to navigate this evolving landscape, understand how to look beyond the hype by considering the essence and the value of each new concept or technology, anticipate its impact, identify the opportunities and the risk, and do the due diligence homework.
One of the concepts I looked into during these data and tech evolutions was MLOps, and I am still studying it, that is how I ran into Marvelous MLOps (great publication by the way). A recent discussion you and I had after I read your article about building the MLOps Dream Team, made me challenge my personal views on the topic and hence wanted to share a complementary perspective.
Raphaël: What challenged your view?
Ahmed: I was surprised by the “radicality” of the article tone, and the “tech — exec opposition” it insinuated. My position on this is: there should be no opposition nor misunderstanding between the top management and the teams as long as we agree that we should align to company strategy, and we collaborate together towards reaching the business objectives.
Raphaël: I will not deny sometimes taking on (too much of?) a radical tone. As techies and developers we can get so frustrated by management + their Strategic Consultancy henchman cooking up roadmaps without involving us, the technical experts, that we can build up some annoyance over time. And let me take this opportunity to share my one of my favorite memes.
Raphaël: But I guess the good thing is, technical expertise involved on the decision-making or not, MLOps seems to be on the rise.
Ahmed: Indeed, with the increasing adoption of Machine Learning within organizations from one hand, as well as the evolution of software deployment cycles through DevOps methodologies on the other hand, we witnessed the emergence of MLOps, which, to make it simple, is the bridge between creating a machine learning model and making it work seamlessly in the real world (production). To be more precise, it is also about automating the deployment process and ensuring maintaining it over time, with short yet secure cycles, and reduced time-to-deploy, aligned with the business expectations.
The opportunity
Raphaël: How is the rapid evolution and rise of MLOps impacting decision-making and strategy level in the C-suite? If it does at all?
Ahmed: The growing integration of Machine Learning (ML) into business operations offers a myriad of opportunities.
From enhancing customer experiences to optimizing supply chains, ML-powered applications are revolutionizing industries, and putting them under industrialized and controlled deployment processes, is key to ensure they deliver the intended value.
But, before jumping into implementing MLOps, it is key to assess and evaluate this opportunity. So, first the question is, is there an opportunity for my organization, and is it fit for my context?
Raphaël: This is a key question for many organizations indeed. I’m listening…
Ahmed: MLOps is without any doubt a combination of two powerful concepts. As we deploy our Machine Learning models, we quickly see the limits of making them “live”. It is not uncommon to have a first deployment (generally a PoC, or Proof of Concept, that goes “Live”), followed by …, nothing. This can be related to many good (and bad) reasons: like the PoC didn’t deliver, or the perceived value was low, but also can be related to the fact that: we build it, now, how to run it? And that we didn’t figure out this part before. MLOps can in this case solve part of the equation, the deployment and maintenance. But, not the business value, that is more intrinsic to the use case and the model.
From a CIO perspective, MLOps offers an opportunity to structure the deployment process and maintenance of the models with strong methodology, but it should not be put in place for the sake of “doing MLOps”, it should serve a clear business purpose in the first place.
But then, is it the only way to do the maintenance? Can’t we just do it “the old way”? Well, I will be a little more controversial on this part.
Raphaël: Haha, I love that. I love some good prodding and poking. Please, roast our MLOps principles and dreams :)
Ahmed: Happily! But also constructively, of course. Let’s say if you are Tesla and you are developing self-driving features: what you are doing is not just about crafting sophisticated ML models. It’s about continuously updating them based on real-world data, ensuring safe and improved driving experiences for users. This continuous loop is MLOps in action, and if you are not adopting MLOps (or some similar practice), you are doing it wrong.
However, if you are simply maintaining a model to predict customers churn, and you have 1 churn per week (so basically a couple of data points to reintegrate in your model every cycle); well, sorry to be disappointing, but MLOps might be an overkill, and with good test (including non-regression testing) and deployment processes, maybe semi-automated, it can be pragmatically sufficient.
We should keep the right balance between the business value and the resources mobilized to deliver that value.
Hence, there is clearly an opportunity, but depending on the context, an assessment is needed before jumping into a full-fledged MLOps approach.
Raphaël: I couldn’t agree more. I think from a data scientist’s perspective, it can be really fun to take on the challenge of learning MLOps (I know not all data scientists will agree here, but in my experience a good part of the DS population is open to it and has the right engineering mindset to take it on). So when the opportunity presents itself it’s tempting to jump right in with loads of new ideas, tools and practices. However, while building bigger ML stacks with pipelines, registries, etc. I have found myself wondering at some points: do we really need all this? Wasn’t our single script + a cron job just as good?
In the movie The Usual Suspects the main character (Roger / Keyser Soze) says: “The greatest trick the devil ever pulled was convincing the world he did not exist.”
Sometimes I think the greatest trick data + ML + AI projects and teams pulled, is that they deliver ROI metrics on everything, except themselves. Insane resources are being poured into the field, and with MLOps we happily ride on those coattails. Is it all worth it? Certainly not always.
The risk
Ahmed: With every new technology or concept, there’s inevitable hype, especially in the tech space. We’ve been here before with Blockchain, Data Science, and even the Internet of Things (IoT).
Raphaël: Let’s not forget Augmented Reality and the Metaverse! 🫠
Ahmed: While these technologies hold vast potential, their real-world applications need strategic consideration. MLOps is no different. Yes, ML can derive unparalleled insights from data, but without an effective MLOps strategy, these insights can remain locked away or worse, be misinterpreted.
With the hype, comes the FOMO (Fear of Missing Out). As a CIO, you do not want your organization to be left behind, and actually, as part of your job, it is your role to ensure that you and your team explore the opportunities that can drive real business value.
Hence, many organizations rush to integrate ML into their operations, driven by fear more than by reasonable thinking. It’s almost akin to a gold rush, where enterprises are rushing to stake their claim without fully understanding the terrain. This reactionary approach can lead to misaligned strategies, where companies might either overinvest in ML solutions without clear objectives or hastily deploy undercooked models, leading to suboptimal results. The flow of solicitations from Sales Reps overpromising on some solutions can also generate a lack of correct assessment of what you exactly need, and the decision is more “techno-push” than “market-pull”.
Raphaël: Again, very relatable. As MLOps engineers we often run into the results of rushed MLOps as well; as a consultant you find yourself arriving at organizations where the wrong tool already has been “pushed” into implementation. Creating inefficient solutions, or even worse, technical debt. As an employee or expert in your own organization, you can find yourself having to fend off suggestions by other departments to implement this or that quick solution they found somewhere. The marketing power of turnkey SaaS solutions for ML platforms, or AutoML for data science, can be a tough cookie to deal with.
So what advice would you give to people on the C-level who have to make decisions on the possible implementation of MLOps, and what risks they should consider?
Ahmed: From my perspective, this comes back to weighing the opportunity and the risk, and balance it with the business value it brings.
Having said all this, and looking beyond the hype, we can then start asking the question: what are the real risks associated with adding an MLOps stream into your teams and organizations?
I believe overhead is the main risk that is overlooked nowadays.
Implementing MLOps processes requires a lot of efforts (and budget):
- you need specific skills so you should manage to recruit the right team members
- you also need specific tools that are suitable to your existing tech stack
- and you need to go through a tough maturity curve to ensure your MLOps processes and tools are reliable and serving their purpose.
To do so, you will need to convince your CEO (or Executive Committee) about the soundness of your approach, your business case and your budget allocation. If you manage a global envelope with a limited budget, as CIO, you should balance what your priorities are for your budget. Typically, if you have gaps in Cybersecurity, that is definitely more critical to fix before moving to new initiatives like MLOps (yes, maybe controversial again). If you have some “margin”, you still need to prove that there is a positive business case.
And here comes the corollary risk: to what extent is it complex to derive ROI from the MLOps approach. How to prove it is viable economically, and that what you are spending is providing gains higher than the spending. ML (& MLOps) projects can often become “black boxes”. Without clear understanding and transparent operations, businesses might pour resources into MLOps projects without tangible returns. Despite significant investment and initial promise, such projects regularly face challenges in delivering consistent value in real-world medical settings.
Raphaël: The stastistics on succesrates of ML and MLOps projects are pretty shocking, with some indicating over a 90% failure rate. I imagine if we take this context into the boardroom, it is going to be an impossible sell.
Ahmed: I believe, if no business case (and ROI) can be proven, it might mean that you should not embark in this journey,
Raphaël: So then… we do nothing?
Ahmed: No.
If you have a valid business case, a clear need, and the “critical mass” to feed MLOps, this is a “no brainer”.
If not, you should balance the risk and start small. MLOps implementations from my perspective are not binary, and you can start developing the initiative internally, maybe with your existing team members, or punctual and cost-efficient support from some contractors and freelancers that can help you kick off the journey. That will help you get a better sense of what you need, refine your business case, and understand the gaps you have.
The idea is not to build a PoC, but to build an MVP: Minimum Viable Process, ready for industrialization.
Why you need to do this: because with the speed at which things are moving, you might need to quickly upscale your capabilities as “the right use case” is on the table, and you do not want to delay the thinking process until when that starts to happen, instead, you should have already anticipated things with your small scale initiative.
I compare this to: have everything ready, so it is just a “push of a button” to have a working plan being executed that can help you quickly and efficiently have working MLOps processes, that you will obviously maintain and improve over time.
Raphaël: C-level decision-making is no walk in the park. A lot of factors to weigh in a complex environment, with a limited budget. Thanks for sharing your thoughts and insights.
Ahmed: My pleasure! And again, just remember that no size fits all, but we need a pragmatic approach, to be adapted to every context and organization. And in all cases, enjoy the learning and the journey!
