Using AI to improve live debates at SXSW
A Dock/Karmarama collaboration threw an AI into the cut and thrust of debate at SxSW
In a world of alternative facts, “fake news”, and psyops, the truth can be an elusive concept.
So could technology and data be used appropriately to help people form accurate and realistic judgments and better understand the world around them?
The Dock, Accenture’s flagship R&D and global innovation center, decided to tackle this question with a high-wire demonstration at one of the world’s most famous festivals. In February and March, a team of AI and Analytics experts helped build an AI to co-moderate a debate between two speakers with opposing visions of the world, at South by Southwest Interactive (SxSW) in Austin, Texas.
This project was a collaboration between the Analytics and AI team at The Dock and creative agencyKarmarama, which is part of Accenture Interactive. The challenge was to use smart data and algorithms to evaluate a live discussion between speakers and to help a human moderator steer the debate towards fair and accurate information.
The team’s goal was to accurately inform the audience about the points being made by the speakers, without them being biased by the ability of the speakers to select, describe and potentially over-index on certain topics. This was also an opportunity to demonstrate and experiment with human-machine collaboration in the Artificial Intelligence domain.
Human + machine debates
This project is indicative of wider industrial trends, where collaboration between humans and technology is helping machines to become smarter and is re-shaping companies and challenging their business and operating models.
The Dock team who took part in this collaboration with Karmarama was comprised of myself, Bogdan Sacaleanu, Christophe Onambele Manga and Andrew Poole, supported by Rory Timlin.
To be disruptive you need to act fast, collaborate in an agile manner and take risks. So in a relatively short time, we developed algorithms and technologies for the prototype for use at SXSW. The core technology showcased at SXSW is based on state-of-the-art natural language processing, plus a set of “ad-hoc” algorithms developed by The Dock.
The full system, orchestrated by the Karmarama team, also includes middleware that manages the output of the speech-to-text components and consumes our API, and a visualization that helps the user to experience the technology.
Rory Timlin, Antonio Penta, Andrew Poole, Bogdan Sacaleanu and Christophe Onambele Manga from The Dock who helped build the AI for Karmarama
The system behind the machine
We designed a process and developed a system for analyzing debates that includes two main stages: the Debate Curation stage, where algorithms, data and human knowledge are used to define what a good debate should be; and the Debate Analysis stage, where algorithms are used in real time to help the human moderator and the audience to evaluate if a debate is aligned with the goals defined in the first stage.
At the Debate Curation stage, machines are used to extract relevant information from a large quantity of data and, in tandem, humans decide what should be considered relevant for the debate.
This process uses two sets of unstructured texts or corpora. The first one, named the reference corpus, is a large collection of articles and news collected by a media agency and/or the debate editors, which has been judged trustworthy. The second is a small annotated corpus, named the editorial corpus. In it, editorial staff select the topics they think should be covered in the debate and used to evaluate its value.
The debate analysis stage is a set of real-time text analytics algorithms and technologies that processes the text from each live speech with the aim of analysing the following information: the breadth of the debate; the accuracy of a debate; the convergence/consensus of a debate; and the themes and sub-themes of a debate.
All this information can be used to evaluate the flow of the debate, and to help shape and improve the discussion towards more relevant topics and possible shared solutions.
The live result
On the day, the event was a huge success. Jon Wilkins, the chairman of Karmarama, moderated the debate and said that the AI added significantly to the event. “When it was time to perform, we learned a lot from our bot. As it followed the debate, it did an interesting job of listening to those themes. And what the debaters discovered was that when you’re in the heart of a discussion, you don’t always realize that you’re plugging into multiple meta-themes. We used the speech-to-text capability to transcribe the debate in real time, from which the AI analysed the themes and highlighted the topic areas. We also trained the AI to recognize phrases and words that indicated the speaker’s mood — whether they were happy, shocked or angry, for example. It also pointed out any dubious or incorrect facts: the effect was as embarrassing as it was educational for our debaters.”
The prototype was an opportunity to prove what kind of role smart data and algorithms can play in the future of the media industry. Running a pilot in a real scenario helped the team to understand the impact of these technologies in a wider context, and to collect feedback from domain experts on what can be done to improve the user experience enabled by our algorithms.
During the project, the team faced a tight deadline and the obvious fear factor of a live on-stage debate. The process threw up a host of fascinating questions, not only on what can be improved in the current solution but also what new ideas should be considered in the next steps. Our backlog is now full of interesting ideas to test, including: incorporating the latest deep learning language models into the system by fine-tuning them with our debates’ corpora; exploring weak supervision techniques to label the data; and exploring advanced computer vision models to measure the flow of the debates by also using the posture and expressions of people.
