Learning By Debating
Interesting research shows that AI agents can master natural language tasks by actively debating.
I recently started a new newsletter focus on AI education. TheSequence is a no-BS( meaning no hype, no news etc) AI-focused newsletter that takes 5 minutes to read. The goal is to keep you up to date with machine learning projects, research papers and concepts. Please give it a try by subscribing below:
(Core ML concepts + groundbreaking research papers and frameworks + AI news and trends) x 5 minutes, 3 times a week =…
Debating plays a key role about how we learn new skills and domains. Think about how much rapidly do you learn something if you are in an environment in which you can express your viewpoints and get immediate feedback. In artificial intelligence(AI) scenarios, most agents are designed to learn in isolation or by feedback from the environment in scenarios such as reinforcement learning. However, the idea of multiple agents debating a task in order to improve their knowledge is largely unheard of.
Why is this discussion even relevant? For artificial intelligence(AI) agents to become mainstream in real world scenarios, they need to master human-like tasks. A natural way to do this, its AI programs to receive human feedback. However, this trivial step is incredibly difficult to achieve as most AI environments are too complex for humans to provide continuous feedback. This interesting learning dilemma explains the fact that, while some AI tasks are too difficult for humans to perform, they can still provide better feedback about the learning process than most AI agents. However, in order to do that, the tasks have to be interpretable from the human cognition standpoint.
Let’s now imagine a world in which multiple AI agents can debate a task to a point that a human judge can provide feedback. The agents will be literally learning by debating and trying to align their interests with the feedback provided by a human judge. AI powerhouse OpenAI has done a lot of interesting work in this space including a recent research paper published by OpenAI that proposes a learning by debating method to improve the training of deep learning systems.
The Learning by Debate technique proposed by OpenAI borrows some concept from game theory specifically in an area known as zero sum debate games. In that type of game, given a question or proposed action, two AI agents take turns making short statements up to a limit, then a human judges which of the agents gave the most true, useful information. The adversarial relationships created in the debate game act as a force to improve the quality of the feedback provided by a human. In the game, one agent will make an arguments, other agents poke holes in those arguments, and so on until we have enough information to decide the truth.
In the OpenAI paper, the debate game can be summarized using this pseudo-algorithm that, in its simplest version, includes two agents competing to convince a human judge:
1. A question q ∈ Q is shown to both agents.
2. The two agents state their answers a0, a1 ∈ A (which may be the same).
3. The two agents take turns making statements s0, s1, . . . , sn−1 ∈ S.
4. The judge sees the debate (q, a, s) and decides which agent wins. 5. The game is zero sum: each agent maximizes their probability of winning
Let’s illustrate the learning by debate method with a simple example from the research paper. Consider, that two AI agents, Alice and Bob which are trying to decide the best place to go on vacation. The opening question of the debate is obviously, “Where should we go on vacation? “ to which the agents respond:
1. Alice: Alaska.
2. Bob: Bali
3. Alice: Bali is out since your passport won’t arrive in time.
4. Bob: Expedited passport service only takes two weeks.
The process continues until we arrive at a statement that the human is able to correctly judge, in the sense that the other agent does not believe they can change the human’s mind with yet another statement and resigns. We do not stop when the human thinks they can correctly judge: after step (2) the human may have thought Bali was obviously correct, not remembering the passport issue; after step (3) the human may think Alaska is correct, being unaware of expedited service. The feedback provided by the human judge is used to produce better decision in future iterations of the game.
The main premise of the Learning by Debate technique is that, in the debate game, It is harder to lie than to refute a lie.
As part of testing the learning by the debate technique, the OpenAI team implemented a an image classifier based on the famous MNIST dataset. The goal of the debate is to predict MNIST digits from 6 non-black pixels, sampled at random for each image presentation when pretraining the judge. Two agents then have a debate where they alternate revealing pixels, stopping at a total of 6 revealed pixels (so the judge sees only a little bit of information in total). One debater is honest and tries to make the judge guess right, the other debater tries to make the judge guess wrong.
Another super cool example is a game that tries to classify dogs and cats images. The OpenAI team published a sample website in which humans can play both judge and debaters and evaluate the performance of the model. Give it a go and have fun learning by debating!