Semantic Parsing using Abstract Meaning Representation
One approach for building a question answering system to answer users’ questions from linked data (or databases/knowledge graphs) is to use a deep learning model with a large application specific training set of question-answer pairs to train an end-to-end system. An alternative neuro-symbolic approach would be to build a system by combining several components each trained on its own generic dataset with very small amounts of application specific qa pairs on the order of a couple hundred or less to adapt to the application domain.
In this neuro-symbolic system, we rely on a semantic parser to parse the input question into a “formal” representation of what is being asked for. In this blog, we present our approach for semantic parsing that uses a broad-domain general purpose semantic representation called Abstract Meaning Representation (AMR); the advantage of this approach is similar to pre-training in end-to-end deep learning models, i.e. to have a generic basis to efficiently use transfer learning to a new specific domain. We apply our approach to QALD-9 [1], a question answering benchmark over linked data based on DBpedia 2016, and show excellent results with minimal amount of adaptation effort. The overall system is described in [NSQA].
We will briefly cover the following topics in this blog:
- AMR overview to explain the richness of the semantics that are captured by AMR.
- Domain adaptation of a generic AMR parser to a domain of interest, namely QALD-9 [1].
- Entity and Relation linking to help ground the AMR parse relative to a given knowledge base.
AMR Overview:
The AMR graph of a natural language sentence is an acyclic graph of concept nodes with directed named edges to argument/modifier nodes. Figure 1 shows the AMR graph for:
“Who is starring in Spanish movies produced by Benicio del Toro?”
The root node has word sense star-01 and is a concept frame that can take 3 arguments as shown in Figure 2. AMR has about 9k concept frames with their argument structure (who did what to whom and when and where) to capture the meaning of general English.
In this example, the AMR indicates the user is looking for actor (ARG1 of star-01 which denoted by amr-unknown) in “movies” from “Spain” (a named entity of type country) and the movie is ARG1 of produce-01; note also ARG0 person named “Benicio Del Toro” is the agent of produce-01.
The general abstract meaning representation is quite effective to capture the “essence” of the meaning of the question. There are a number of approaches for building an AMR parser to produce the graph for a sentence. We, at IBM Research AI, have developed one approach using a deep learning model using a stack of actions to create the graph. Our AMR parser establishes a new state-of-the-art performance on traditional LDC test sets [2].
Note that several paraphrases of a sentence will have the same AMR graph as
shown in Figure 3. For example, the following sentences have the same AMR. Note that boy is ARG0 of want-01 and ARG1 of believe-01 and girl is ARG0 of believe-01 which is ARG1 of want-01.
Domain Adaptation:
To improve the AMR parser performance on the genre of “questions” as compared to declarative sentences which are predominant in the LDC AMR corpus, we added training data from human generated AMR graphs for questions from Question Bank, SQuAD, and LC-QuAD 2.0. We compared 3 AMR parsers: i) a parser trained on the original LDC AMR corpus (AMR 2.0), ii) parser trained on a mixture of LDC and Questions, and iii) the parser also fine-tuned on 350 question-answer pairs from QALD-9 domain. On the QALD dev set the pre-trained AMR parsers on LDC and LDC+Questions achieve a Smatch score of about 80% and 86% respectively (Smatch is roughly an F-measure on the triples node-relation-node in the AMR graph). Adding question AMRs is very helpful for the pre-training phase. In addition, when we perform a fine tuning on the 350 QALD-9 sentences for domain adaptation, Smatch improves further to 90% (examples of the fine-tuned parser output is shown in Figures 2 and 3 above.)
Entity and Relation Linking:
To make the AMR parse actionable we need to do entity linking to map mentions of entities in the question to the KG; e.g. that Benicio is a particular person in the KG and Spain is a particular country in the KG. We have used an off-the-shelf SOTA entity linking system described in detail in this paper [3].
In addition to entity linking, we also need to do relation linking that maps AMR concepts such star-01 and produce-01 to the corresponding DBpedia (2016) ontology relations dbo:starring and dbo:producer. The relation linking module achieves SOTA performance on various RL benchmarks as discussed in [4, RL].
The AMR parser, entity linking, and relation linking components are combined with two additional modules: i) AMR-to-Logic which convert the grounded AMR into a SPARQL query and ii) Logical Neural Network (LNN) perform additional reasoning to improve the answerability of the SPARQL query. The overall system of systems, shown in Figure 4, is discussed in more detail in Neuro-Symbolic QA blog. The Neuro-Symbolic QA system (NSQA) has achieved a Macro F1-QALD of 45.3% on QALD-9 and an F1 of 38.3% on LC-QuAD-1 [5], outperforming state-of-the-art systems tuned on the respective datasets (43.0% and 33.0%). A detailed description of NSQA is available in [6].
To summarize we have developed NSQA, a system that does not need end-to-end fine tuning, yet outperforms the state-of-the-art systems on KBQA tasks.
References
[1] Ricardo Usbeck, Ria Hari Gusmita, Axel-Cyrille Ngonga Ngomo, and MuhammadSaleem. 2018. 9th Challenge on Question Answering over Linked Data. In QALD at ISWC. 58–64.
[2] Young-Suk Lee, Ramon Fernandez Astudillo, Tahira Naseem, Revanth Gangi Reddy, Radu Florian, and Salim Roukos. 2020. Pushing the limits of AMR parsing with self-learning. In Findings of EMNLP.
[3] Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. Scalable Zero-shot Entity Linking with Dense Entity Retrieval. https://arxiv.org/pdf/1911.03814.pdf
[4] Nandana Mihindukulasooriya, Gaetano Rossiello, Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Mo Yu, Alfio Gliozzo, Salim Roukos, and Alexander Gray. Leveraging Semantic Parsing for Relation Linking over a Knowledge Base. 2020. The Semantic Web ISWC 2020. 402–419.
[5] Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs. http://lc-quad.sda.tech/lcquad1.0.html
[6] Pavan Kapanipathi, et al. 2020. Question Answering over Knowledge Bases by Leveraging Semantic Parsing and Neuro-Symbolic Reasoning. https://arxiv.org/abs/2012.01707
