Machine Reading Comprehension

By Siva Sai N, SVP AI Product

As intelligent conversational interfaces (Chatbots) go mainstream, the expectation from this medium has grown to expect near human-like ability to answer queries, transact and resolve issues for an enterprise.

Answering factoids has remained a big challenge in conversational interfaces today. The state-of-the-art today use simple Question Answering Systems that are trained with curated questions and answers by business users. At Active.AI we reinvented questions answering systems via award winning “Cognitive Q&A” module which uses combination of deep learning and reinforcement learning to achieve ability to paraphrase, identify query focus and find the best of possible answer even if the variant wasn’t provided by the enterprise users.

The limitation however is that the factoid must exist in the training data (a curated set of question-answer Pairs) prepared by the enterprise users. Given the range of products and services that large enterprises like banks offer, the sheer number of questions that need to be trained to ensure coverage is immerse. This also presents a maintenance challenge as products, conditions and regulations constantly change and these need to be propagated to yet another channel that is much more complex than a simple CMS driven website.

Reading comprehension is an AI-complete task, which requires a Q&A system to process a piece of text, comprehend and be able to extract the span of text which is the answer to the user query.

Stanford Question Answering Dataset (SQuAD)is one such public challenge pushing the limits of span prediction type of Question answering mechanism. (

SQuAD 1.1 test for machine reading comprehension (MRC), creating a system that could read a document and answer as well as a human. SQuAD 2.0 was to “encourage the development of reading comprehension systems that know what they don’t know.”

We have been investing in this area for the last 2 quarters and our novel Approach to solving Reading comprehension is based on multi task learning of two Natural language tasks :

1. ARSG :Anticipated response structure generation i.e. Anticipate different response structures ,given a query even without knowing the answer.

E.g. for a query “what is your age” , system can generate multiple anticipated response structures e.g “my age is …….”, “I am …… years old”

2. Predicting the span of the text/document which holds the answer.

The system is trained to learn both the tasks together , sharing vital information across the tasks for a better question answering.

We have submitted a part of the core solution to SQuAD and we have made it to the Leaderboard of Stanford machine comprehension test. Active.Ai is currently featured above Microsoft Research Asia, Microsoft Business Applications AI Research, IBM Research AI and Allen Institute for Artificial Intelligence. We are also among the Top 10 worldwide on SQuAD 1.1 dataset and Top 20 worldwide on SQuAD 2.0 dataset, with the top scores of 74.746 (exact match) and 78.227 (f1), on SQuAD 1.1, very close to the Human performance cited on this dataset, at 81.30 (EM) and 88.9(F1). This was our very first submission and we are delighted by the results. We strive to better this performance over the next 2 quarters.

Stanford Question Answering Dataset (SQuAD) Leaderboard

While SQuAD ranking is one thing, we are more interested in the real world deployment of this technology. SQuAD is relatively much simpler task than the real world problem because we need to process a lot more content for any practical application in conversational interfaces, which makes the ability to select candidate paragraphs from over 10,000 paragraphs very important. Along with the ability to understand structured embedded content like Tables important as well.

General purpose reading comprehension systems are still extractive in nature. As in, it can only answer if the answer is present directly in the content that it was trained on. Ability to Infer facts from multiple parts of the content and formulate a response to users query is exponentially more complex.

We foresee that our first deployment on reading comprehension will happen in 2019 and will most likely be focused on an internal facing application to help the enterprise make better decisions and provide better service.

To find out what exciting innovation the team is working on, please visit Active.Ai.

Active AI