Yet another unhealthy machine learning project
This blog is about the experience of working on a machine learning project in the research phase. The research phase might be repetitive and boring, without any reward at the end of a long tunnel. However, working on the research can be fun as well. It allows our team to shine with creativity and collect a reward after taming the dragon. The dragons in the blog are not machine learning models. Nor are any tech-related difficulties. The dragons are in our minds. I’m talking about missed expectations, blame, finger-pointing, arrogance, ignorance, resentment, and other destructive feelings festering in the long-tortured brain.
Unicorn project
Let’s meet our stakeholders. They are non-tech people who heard that machine learning could solve anything, and it is a way to attract investments. Their thoughts: “Everyone is doing it, so why not? Let’s go for 99% accuracy. No one in the industry has what our unicorn project gives. We can get it in a month by paying an ML guy to do it. Let’s promise high accuracy to a big champ with money. We don’t need to consult our guy. He can do it.”
Let’s meet our unfortunate ML hero John. He got an opportunity to work on this fascinating machine learning project. It is a dream job that sounds magical and sparks unicorns every time someone mentions its name. However, as the project approaches a deadline, the pressure and emotions start bubbling.
- You said it’s 95% accurate, but why it doesn’t work for this case?
- Hey, I notice it doesn’t work here.
- Can you fix this case?
- Let’s hop on a quick call, I want to understand….
- We need to do this ASAP. The stakeholder wants it…
- Yesterday it worked here, but now it has 96% accuracy but doesn’t work after retraining
- You said it will work after fixes…
- It’s urgent to fix this…
- Let’s hop on a quick call, I want to understand….
- Accuracy is 95%, well, it’s not **that** terrible…
- Can you fix it now?
- You fixed it last month, but it’s not working now, again.
- It works, barely…
- When can I test it?
Months have passed, deadlines failed out of fear that something wouldn’t work as expected. Everyone started to point fingers at one another. The age of rationality has passed, the age of destructive emotions has come. The festering resentment started to sprout inside heads. Shortcut, after shortcut, the journey has been lost. What is left is only a trace of an original goal. The atmosphere became unbearable. In the end, after struggling, the project has failed. The funding was cut. What went wrong? Who is to blame, the evil stakeholders or our poor John?
Neither John nor stakeholders are evil in their hearts. No one here is an asshole. I would argue that everyone wanted the project to succeed. Everyone worked towards the same goal, yet they have failed. Everyone has unmatched expectations and perspectives of what is right.
Expectations were like fine pottery. The harder you held them, the more likely they were to crack.
- The Way of the Kings, Brandon Sanderson
There is no point in blaming one another. Blaming and finger-pointing would make everyone defensive and distrustful. It would spread bitterness, resentment, and cynicism like a disease. Instead, all of them learned something. John has learned that it is not enough to stay in his dark cellar researching without communicating results and options. Stakeholders have realized that machine learning brings a hell of uncertainty to the project. They also need to be thoughtful about what research means. Even though the project has failed, it opened the door to new challenges and possibilities. Now, they see what structure they can form out of the chaos that almost consumed them.
Endnote
I trust everyone can relate to this story, as either John, stakeholders, or either someone who worked with John. It was a painful experience, though, but we learned something out of it. We learned that we need to overcome our arrogance, pettiness, cynism, and face the dragons inside our heads.
In the next blog post, we will dive into what John has done and what could be different.
All blogs from the series:
- Yet another unhealthy machine learning project
- Understand the domain and data before starting a machine learning project
- The evaluation metrics and error analysis for improving the process in the research phase
