A $10,000 mental experiment for those who run Machine Learning projects
Research projects are fundamentally different from development projects. In development, you pay for tangible results. In research, failure can also be a valuable outcome. What can ease the pain of losing money through failed attempts?
by Artemy Malkov, PhD
Are you willing to take $10,000 out of your own pocket and give it to someone, only to have them spend it and fail the next day?
Most research projects, including ML/AI projects, take more time and end up being more expensive than originally planned. This is easy to understand (the trial-and-error process always has a decent percentage of failures), but hard to afford. Business is always about maintaining efficiency and reducing unnecessary costs.
Is failure a necessary cost?
The alternative is to do nothing and cut your research budget to zero. However, in this case you risk lagging behind the competition and losing the entire market within several years. So, companies have to take risks and sponsor innovation and research projects.
As machine learning and AI gain in popularity, more and more companies are pursuing AI initiatives. Here are some ideas for those who are aware of the gaps in their process, and want to establish mature and lean ML / AI product management.
Once you become an AI product manager, you become that “someone” who receives a $100,000-$1,000,000 check from your company’s pocket (which may feel the same as $10,000 from a personal pocket) in order to show tremendous success. Unfortunately, chances are, failure is a more probable outcome.
AI is not similar to traditional software development
If you are running an AI / ML project, it’s obviously not your first project. You have enough experience, with most of it probably being in the IT field. You know about managing teams, creating backlog, writing specs, etc.
However, there’s something different about AI / ML projects that sometimes makes you feel like you’re in a minefield. Data scientists almost never keep to their deadlines. They promise you something, saying, “I need a couple of days to check this and that.”, and after a week they say, “It didn’t work out. I need a few more days.” The same result after another week and so on. In the end, everyone is frustrated — your sponsors say, “We see no progress!”, your team says, “We work day and night and need more time!”. Meanwhile, you’re the manager in the middle between the devil and the deep blue sea, responsible for a project, day after day looking like a failure and a lost investment.
Even if your project is still going fine (congratulations — you’re a great leader), there’s no guarantee you’ll never face a situation like that. A lot of ML projects get stuck when there isn’t sufficient data to fuel the models, and when the expectations of accuracy are too high, beyond the state-of-the-art level and human-level of performance.
After all, machine learning is driven by statistics. Therefore, every success and failure is the outcome of some sort of gamble.
A research project is in fact a process of turning questions and uncertainties into validated hypotheses and discoveries.
Research is a maze game with multiple blind alleys. If you are lucky and pick the right pathway in the labyrinth, you save a lot of time and money. If not, your team may spend weeks, months, or even years, trying to solve a puzzle, construct a working solution, hit the target score of an ML model, or whatever else is your project about.
Organize your next research project the right way from the beginning.
There is a nice M.I.S.S.I.O.N. framework that can help you set up an AI project the right way. In short, here are some key points:
- Research (discovery) projects are much harder to estimate than traditional development (delivery) projects. Development: deliverables may be planned quite precisely, and failures are unwelcome. Research: discoveries are unpredictable and occur after a sequence of failed experiments, and failures are inevitable by design.
- You can not predict whether the next experiment will fail or succeed, but you may manage the risks and expectations of the stakeholders, so that even if it fails, it will be a valuable learning experience and a step towards success.
- Don’t rush with machine learning experiments. First, establish a solid M.I.S.S.I.O.N. statement, achieving clarity and a shared understanding about the main components — Money, Ideas, Strategy, Skills, Inputs, Outputs, and Nuances of the upcoming AI project.
- Without clarity on each component, you are taking a huge risk. If something goes wrong down the road (and it will, because research projects are failure-driven), you may lose the support of the stakeholders.
- Spend additional time early in the project in order to establish a transparent, failure-tolerant, yet propulsive management process.