Applied Machine Learning: New tech, but ‘same same’ in implementing it successfully for Temasek
How did we overcome our challenges specific to building a ML product in a business setting? The answer might be familiar to you.
This is the story of our first ML product rolled out in Temasek last year. Stay tuned for more stories of our work!
When I first joined Temasek as an early member of the Root Access team, our first order of business included meeting people across the organisation to understand their daily work, in order to figure out how digital technology could help them.
Picking our first business use case
While making our rounds, one business use case stood out. A team in Temasek helps to govern the investment metrics that feed into the due diligence process of deciding whether to make an investment. These investment metrics have to be generated from a list of companies classified into the appropriate industries.
Typically, this list is first obtained from Bloomberg. Then, the team does further due diligence in reviewing this list and manually sieving out inappropriately classified companies to meet internal standards. The industry classification of a company can be a subjective and tricky matter. For example, it is possible for companies to have businesses in multiple industries.
Every year, the team sets aside a total of 38 man days for this review exercise, but within that timeframe, they focus only on checking a subset of industries more significant to our business. Ideally, the team would want to do a full review of all industries, but the value-to-effort ratio would be too low.
This was the perfect use case to consider building a Machine Learning (ML) solution for the first time in Temasek — the goal of classifying companies into industries is clear and straightforward, there is a trove of readily available datasets accumulated across the years, and above all, the productivity gains would be tremendous. In fact, a ML solution would bring new value in performing due diligence across the entire exhaustive list of industries and companies, beyond just a subset.
Our biggest challenge: Getting business stakeholders to trust the model
Solving problems using ML is not without its challenges. Besides the usual criticisms such as the opacity of deep learning etc., there are challenges peculiar to solving business problems in the real world using ML:
- Firstly, there is no such thing as “100% accuracy” in ML problems (at least if that happens, it’s most likely due to overfitting). How can we move our business colleagues away from getting too caught up in the accuracy score? What’s considered a “good enough” accuracy level for the purposes of the business use case?
- How can we define the accuracy metric in a way that gives our business colleagues real assurance in the model?
- How much data do we have to manually label? If we need our extremely busy business colleagues to help with that because the domain knowledge lies with them, how do we convince them to spend the time needed? How could we, as the tech team, come to acquire the domain knowledge needed too?
The root issue is this: How can our non-technical business colleagues come to trust the output of a new ML model?
The root issue is this: How can our non-technical business colleagues come to trust the output of a new ML model?
Let me take you through how we grappled with one of our challenges.
Case in point: Pivoting on the “accuracy metric”
Our solution strategy was a supervised model approach with two major iterations. In the first iteration, we experimented with an ensemble of models, which hit an average accuracy level in the high 80s. Because we prioritised hitting high accuracy scores, our approach ended up having huge implications on the computation pipeline, in terms of time and resources needed to do model training and deployment.
In our second iteration, we embarked on building a new ensemble of models, which turned out to work quite well. We also managed to reduce complexity of training time and deployment by many folds.
However, during the second iteration, after a review of the business requirements, our business colleagues started expressing concerns over the validity of the initial test set used — which in turn affects the definition of the accuracy metric. By this time, we were actually more than halfway through the project.
However, we immediately assured our business colleagues that it was fine to redefine the test set, as long as it was to meet critical business requirements. This change in requirements meant that we had to put in extra work to tweak the model accordingly. But as a team, we are committed to putting our users first and solving the right problems for them.
Our business colleagues ended up putting in some extra hard work too. Since the domain knowledge lies with them, it made sense for them to be the ones labelling the new data for the test set. In fact, they even volunteered to spend their weekend on this in order to expedite handing over the test set to us. By this point, our business colleagues were clearly invested in the problem-solving process together with us; the good output from the first model iteration had given them confidence that ML can work for them.
With the new test set and accuracy metric, after further training and fine-tuning, our new model had managed to hit an even higher average accuracy level than before.
The key to applying ML successfully is people (and good ole Agile)
Because of the challenges associated with solving problems with ML in a business setting, the success of the first ML product in Temasek was arguably more contingent upon the close collaboration with our business colleagues, rather than the technical problem-solving process itself. As engineers, we can tend to geek out over the intricacies and technicalities of our solutions, and that’s a great culture to build within a technical team. But if our business colleagues cannot bring themselves to trust the model in the end, they simply would not make use of its results. What good would our fancy ML techniques be then?
This highlights the importance of delivering products in an agile manner. For the business team to come to trust the ML model, they must first come to trust the tech team as people. Agile creates a consistent cadence for people to spend face-time with each other, which gives both tech and business teams the opportunity to build trust. In order to solve problems effectively, it is important to create a safe space for everyone’s doubts and issues to be quickly surfaced and addressed.
For the business team to come to trust the ML model, they must first come to trust the tech team as people.
Agile has been around for a long time (the Agile manifesto was written in 2001), and it was first invented for “conventional” software development. This experience also showed that Agile is very much applicable to delivering Machine Learning solutions, especially for first runs — even if Machine Learning may have a different problem-solving paradigm from traditional software engineering. In fact, we ended up adapting conventional Agile practices to suit building a ML product better (this probably deserves another blog post of its own).
The original Agile philosophy is designed to embrace uncertainty, and I’ve come to appreciate its power to help in building a new product team figuring out a new technology. By working in an effective agile manner, we got to a point where we could collectively commit to a decision during a 10-minute daily standup to redefine a brand new test set, when we were already more than halfway through the project — no muss, no fuss, just a shared dedication to making things work.
We owe the success of this product equally to our wonderful, open-minded business colleagues willing to try a new way of solving a long-standing, existing problem. This experience has only reinforced for me Temasek’s appetite for innovation.
More links to check out:
- 5 Lessons Learnt from Starting Machine Learning Projects in the Enterprise written by Pier, one of my teammates
- A Tech In Asia article published last year on how Temasek Root Access has been helping to drive Temasek’s digital transformation — this story of this product was featured there too
