Sharing, without oversharing, in collective machine learning
Researchers find a way to harness the power of collective intelligence, while still respecting privacy and proprietary knowledge
When you live in a tech-driven world like ours, it’s hard not to be bombarded with buzzwords like big data, artificial intelligence, and neural networks. Machine learning is another favourite — you’ve probably heard of it, but what about collective machine learning?
Computer scientists call the process of feeding computers statistical models and algorithms machine learning. The goal: to train computers to identify patterns from data so that they can eventually act without a human being telling them explicitly what to do. Collective machine learning takes this to another level, by combining multiple machine learning models together to harness their collective intelligence. The desired end product is a single predictive model that is more accurate than the sum of its parts.
“Collective machine learning combines multiple machine learning models together to harness their collective intelligence. The desired end product is a single predictive model that is more accurate than the sum of its parts.”
“Improving predictive capabilities is the underlying aim,” says Bryan Low, an assistant professor at the NUS School of Computing and principal investigator at the NUS Centre for Research in Privacy Technologies (N-CRiPT), who studies machine learning and multi-agent systems. There are many instances where collective machine learning could come in handy, he says. For example, autonomous vehicles coming from opposite directions could “talk” to one another about traffic and road conditions. Or doctors from different hospitals could combine information from their patients’ medical records to identify trends in how a disease may progress within a certain population.
Because machine learning works on the fundamental premise that the more data available, the better, it’s important that information be shared regardless of which company manufactured the autonomous vehicles, or which healthcare group the doctors belong to. But equally important is the need to respect information that is sensitive, private and proprietary.
Building the Tower of Babel
These conflicting needs of sharing versus privacy protection leads us to the model fusion problem: collective machine learning is beneficial, but how do we combine different models that are essentially black boxes and heterogeneous in nature? In other words, how can we design a single predictive model out of all the different models, without knowing their internal architecture and local data, and when they are all so differently designed?
Prof Low likens the problem to building the Tower of Babel. According to biblical literature, people living in Babylon wanted to build a tower tall enough to touch the heavens as a symbol of how great they had made their nation. Disliking the pride and arrogance of the Babylonians, God decided to thwart their efforts by suddenly causing them to all speak different languages, instead of a single one that previously united them.
“It’s like having four prophets of different races and languages,” says Prof Low. “They can’t speak to one another, but somehow we want to fuse their predictions to get a common consensus of whether the Babylonians will still be successful in building their tower.”
To overcome this challenge, Prof Low and his collaborators from Carnegie Mellon University and the MIT-IBM Watson AI Lab in the U.S. have built a collective learning platform — what they believe is the world’s first to tackle the heterogeneous and black box problems that model fusion presents. The researchers proposed two fusion methods — CIGAR for more light-weight combinations, and COLBI for heavier ones — that work by finding a common language to unite the disparate models.
To further explain, Prof Low returns to his Tower of Babel analogy. The prophets are like the different proprietary machine learning models, unwilling to divulge their trade secrets, he says. But each prophet has an apprentice who can speak a common language. “We can think of these apprentices as surrogate models for the prophets,” says Prof Low. “They do the translation and we then combine these surrogates into one consensus model to come up with a prediction.”
Similarly, the researchers’ solution to the model fusion problem involves creating a collective fusion paradigm that learns the predictive behaviours of the composite models. The information gleaned are succinctly encoded into information summaries which are then used to derive a single predictive outcome.
When the team published their findings, many people responded with surprise, says Prof Low. “They didn’t think you could form a surrogate model from the different machine learning models — it was unheard of,” he says. “But people found it exciting because being able to do that opens up the possibility of what they can do in the future.”
