Introduction to Project Debater APIs
Project Debater team authors: Roy Bar-Haim, Yoav Katz, Elad Venezian, and Noam Slonim
Introduction
Information on the web is overwhelming. No one person can read every article, every review, every piece of information that could be relevant to a topic. Developing tools that allow data scientists and other teams to boil the overflowing pot of text data down into a manageable amount of information that is not only consumable at a human level, but also organized into distinct main points would be a dream for those who need this, such as consumers in need to make a quick choice, but one they can feel confident in, product managers needing to decide on a tool or approach for a large project, or public agencies seeking understanding from large freeform municipal surveys. Building on years of research at IBM as part of the Project Debater, the Project Debater API (currently available in an early access release) gives data science teams the ability to quickly build well-crafted text summarizations that allow humans to comprehend a large scale of information with the important details included and the fluff dropped out.
Project Debater
Project Debater is the first AI system to successfully engage with a human in a live debate. Project Debater made its debut in 2018 in San Francisco and was featured in several live debates since then, including a 2019 debate with 2016 World Debating Championship’s Grand Finalist and 2012 European Debate Champion Harish Natarajan.
To debate humans, an AI must be equipped with certain skills. For example, it has to be able to pinpoint relevant arguments for a given debate topic in a massive corpus, detect the stance of arguments and assess their quality. Given the complexity of the task, and the lack of end-to-end training data, the problem could not be addressed by a single monolithic system. The Project Debater team broke the grand challenge into a set of modular problems to be approached in parallel. Beginning in 2014, the team has regularly published research on argument mining, argument quality, expressive text to speech and more recently key point analysis, which we discuss below. By solving these problems separately to build a greater overall system, the research team built an effective debater system capable of going head-to-head with an expert human debater.
Project Debater Early Access Program
Many of the underlying technologies that enabled the Project Debater event, as well as more recently developed ones, are now available as software services for both academic and business use.
Overall, we released 12 APIs, which can be divided into three groups:
Core NLU Services
These services include efficiently identifying mentions of Wikipedia concepts in text, determining the relatedness between such concepts, short text clustering, and common theme extraction for texts. These general-purpose services can be used in many different applications.
Argument Mining and Analysis
These services include detection of sentences containing claims and evidence, detecting claim boundaries in a sentence, argument quality assessment and pro/con stance classification.
Content Summarization
Narrative Generation and Key Point Analysis are two high-level services that create different types of summaries. When given a set of arguments, Narrative Generation constructs a well-structured speech that supports or contests a given topic, according to the specified polarity. Key Point Analysis is a unique summarization paradigm that includes an important quantitative angle. This service summarizes a collection of comments on a given topic as a small set of automatically extracted, human-readable key points, each assigned with a numeric measure of its prominence in the input. It can be applied to different types of opinionated texts, such as user reviews, customer feedback, survey responses and posts in social media.
In addition to these APIs, the Debater Early Access Program includes the Speech by Crowd application that allows creating a pipeline of these technologies via a Web UI , with no programming required.
Key Points Analysis and Narrative Generation were recently demonstrated in the “That’s Debatable” television series and in the “Grammy Debates with Watson” backstage experience, where they summarized pro and con arguments contributed online by thousands of people, discussing debate topics ranging from social questions to pop culture.
Key Point Analysis: Real World Applications
Let’s take a closer look at how Key Point Analysis can be applied to solve real-world problems. As an example, consider the community survey conducted by the City of Austin, Texas in 2016 and 2017. the respondents were asked “If there was ONE thing you could share with the Mayor regarding the city of Austin (any comment, suggestion, etc.), what would it be?”, and over 3,000 responses were collected. The table below shows some of the top key points extracted from this dataset, and for each key point — the percentage of matching comments and two examples of such matches.
The example illustrates several important advantages of key point analysis:
- Many of the current methods for analyzing textual collections, such as topic modelling or key phrase extraction, aim to find salient words or phrases, which often do not provide sufficient detail for understanding the main points being discussed. In contrast, key point analysis provides human readable, concise statements, which are more informative.
- Existing multi-document summarization methods provide text-only summaries that lack a quantitative dimension. Key point analysis quantifies the prevalence of each key point. For example, we see that 4% of the respondents asked for more parks, walking and biking trails.
- Each key point is linked to its matching comments, providing convenient, hierarchical organization of the data. Matching is done at the semantic level, and is able to detect different phrasing of the same ideas.
We also applied key point analysis to a large-scale dataset of consumer complaints collected by the Consumer Financial Protection Bureau. Below is the resulting summary:
The top six key points are shown. Each key point is a human-readable, concise statement. The summary also contains for each key point the number of matching comments and their percentage in the data. It is also possible to examine the comments that match a specific key point in order to find additional nuances on that specific topic. The coverage achieved with the top six key points is 51%; in other words, 51% of the comments are mapped to these key points. We can increase the coverage by adding more key points.
Another real world example of applying Key Point Analysis at scale is the analysis of the 2020 IBM Employee Engagement Survey. Extracting insights from free-text responses submitted by over 300,000 employees would be a challenging and time-consuming task even for experienced data scientists. Using Key Point Analysis, IBM HR data scientists were able to extract actionable insights and provide high quality summaries to executives with much greater effectiveness than with standard approaches such as clustering, topic modelling or sentiment analysis.
The HR data scientists and our team analyzed more than 550,000 sentences. The sentences were split into positive and negative sets and Key Point Analysis was performed for each set separately.
Next Steps
The Project Debater APIs are currently available freely for academic use and also for commercial use, with a free evaluation period.
A practical hands-on tutorial demonstrating how to use Key Point Analysis and other Project Debater services for analyzing surveys is available We encourage you to go through the tutorial on GitHub and then try the APIs out on your own use case, or read our tutorial piece on how to use Project Debater with the Austin municipal survey data as discussed above.
The Project Debater team is planning to share their work and experience in a few formats in the coming weeks. If you want to learn more, please follow This Week in Machine Learning for an upcoming appearance by a Debater team member, and follow AI Camp for upcoming meetup announcements on Project Debater. Finally, here on Medium, we will be highlighting use cases demonstrating the potential of the Debater API with key point analysis research and potential industry use cases including using Key Point Analysis for analyzing user reviews and social media. Stay tuned!
