Can Machine Translation be a Reasonable Alternative for Multilingual Question Answering Systems over Knowledge Graphs?
Spoiler alert: yes it can!
TLDR
Providing access to information is the main and most important purpose of the Web. Despite available easy-to-use tools (e.g., search engines, question answering) the accessibility is typically limited by the capability of using the English language.
In this work, we evaluate Knowledge Graph Question Answering (KGQA) systems that aim at providing natural-language access to data stored in Knowledge Graphs (KG). What makes this work special is that we look at questions in multiple languages. Mainly, we compare the results between the native support quality values and the values obtained while integrating machine translation (MT) tools. The evaluation results demonstrated that the monolingual KGQA systems can be effectively ported to the most of the considered languages with MT tools.
The Problem
Given recent statistics, only 25.9% of online users speak English. At the same time, 61.2% of the web content is published in the English language. Therefore, the users not capable of speaking or reading English have only limited access to the information provided on the Web. Hence, despite the disputed statistics, a clear gap of the information accessibility exists on the Web. This gap is referred to as the digital language divide.
Nowadays, whenever a user queries a search engine they expect to receive a direct answer. The direct answers functionality is based on the Question Answering (QA) methods and driven by the knowledge graphs. In 2012, the Google Knowledge Graph was presented that powered Google’s direct answers. For example, when one asks Google “Who wrote Harry Potter?”, the following structured result is expected (see Figure below).
These direct answers are more readable and satisfy the user’s information need right away, s.t., they don’t need to open each “relevant” web-page given by the search engine (as it was done before) and search for the requested information manually on the opened web-page. But if we ask the same very simple query in a low-resource language — Bashkir (Turkic language with 1.4 million speakers) — the results are not really satisfying (see Figure below).
Thus, there is an obvious limitation of knowledge accessibility on the Web that is visible for the people that don’t speak any “popular” language on a sufficient level. Therefore, in this article we address this problem in the context of Question Answering over Knowledge Graphs (KGQA) following the question “Are automatic machine translation tools usable to increase the accessibility of the knowledge on the Web for non-English speakers?”.
Approach
To answer the research question above, we conducted a large evaluation study. During our experiments, we follow a component-oriented methodology by reusing off-the-shelf QA components to gain insights for the defined topic. As our goal is to evaluate KGQA systems that are adapted to an unsupported language via machine translation (MT) tools, we therefore need:
- A set of multilingual KGQA systems supporting particular languages.
2. A set of high-quality questions written by native speakers in different languages (where two groups are required: one that is supported and one that is not supported by existing KGQA systems).
3. A set of MT tools that are able to translate questions given in unsupported languages into the supported ones.
For the first point, we used the following well-known systems: QAnswer, DeepPavlov KBQA, and Platypus. For the second point, we used our QALD-9-Plus dataset that has high-quality questions in English, German, Russian, French, Ukrainian, Belarusian, Lithuanian, Bashkir, and Armenian languages and the corresponding answers represented as SPARQL queries over DBpedia and Wikidata knowledge graphs. For the third point, we used Yandex Translate (commercial MT tool) and Opus MT models by Helsinki NLP (open source MT tool). The evaluation was done using the GERBIL platform. Hence, the following experimental setup was built (see Figure below).
Results
From the experimental results, we clearly observe strong domination of English as a target translation language. In the majority of the experiments, translation of a source language to English gave the best QA quality results (e.g., German → English, Ukrainian → English).
In the very first case, where English was the source language, the best QA quality was achieved on the original (native) questions as using machine translation additionally was decreasing the QA quality (see Figure below).
Only in the case where Lithuanian was the source language, the best target language regarding the QA quality turned up to be German (i.e. Lithuanian → German), while English also demonstrated reasonable quality (i.e. Lithuanian → English). Although the experiment was carefully designed, we considered this case as an outlier. Nevertheless, such outliers might have a significant impact while improving the answer quality of QA systems.
Summary
Our main conclusion is that machine translation can be efficiently used for the purpose of establishing multilingual KGQA systems for the majority of languages. It turns out that the most optimal way of using MT tools is to just translate the source language (e.g. German, Russian, etc.) into English — this will result in the highest quality question answering process. Hence, even if the source and target languages are from the same group (e.g., Ukrainian → Russian — slavic group) it is better to translate them to English from the quality point of view.
Despite our results and the concluded recommendation to extend the QA system by machine translation component, we would like to point the research community to many open questions that might affect the answer quality. Therefore, we plan to extend our experiments with additional languages. We welcome any input to help us expand this research.
If you want to see more detailed results, please see our recent paper: https://dl.acm.org/doi/10.1145/3485447.3511940
The video presentation is available here:
Acknowledgments
I would like to thank the co-authors of this work, namely: Andreas Both, Dennis Diefenbach and Axel-Cyrille Ngonga Ngomo.
Authors of the paper would like to thank all the contributors involved in the translation of the dataset, specifically: Konstantin Smirnov, Mikhail Orzhenovskii, Andrey Ogurtsov, Narek Maloyan, Artem Erokhin, Mykhailo Nedodai, Aliaksei Yeuzrezau, Anton Zabolotsky, Artur Peshkov, Vitaliy Lyalin, Artem Lialikov, Gleb Skiba, Vladyslava Dordii, Polina Fominykh, Tim Schrader, Susanne Both, and Anna Schrader. Additionally, authors would like to give thanks to Open Data Science community for connecting data science enthusiasts all over the world.
