What is a Neural SPARQL Machine?
The Web contains a tremendous amount of information, which can be found in different formats, shapes, and languages. Back in 1999, Sir Tim Berners-Lee — also known as the inventor of the Web — had a dream to make the Internet ‘understandable’ by machines. At the time, his vision for a Web of Data was largely criticised by researchers all around the world. Today, 21 years later, most of the Web is still unstructured (in the form of text, images, videos), however recent advances in machine learning showed to be able to extract precise concepts from nearly any medium. This has contributed in restoring the hype for semantic networks, abstract representations that frame entities and the relationships among them.
There is now a general consensus that such networks, when backed by a common schema, are referred to as knowledge graphs. Despite the fact that knowledge graphs can rely on different frameworks, the official query language and data model of the Web of Data are, respectively, SPARQL and the Resource Description Framework (RDF), which build upon the principle that everything can be described using statements composed of a subject entity, a relationship (or predicate), an object entity, and a context.
“The key idea is to consider SPARQL as another natural language and employ machine translation techniques for data access.”
A Neural SPARQL Machine (often abbreviated as NSpM) is a software architecture based on deep neural networks that lets humans retrieve information from a knowledge graph using solely their natural language. Our mission at Liber AI is to create a universal approach to automatically generate an artificially intelligent ‘oracle’ able to identify the user’s intent and provide them with the desired information. We started this journey with the publication of our first paper SPARQL as a Foreign Language¹ in 2017.
As suggested by the paper title, the key idea is to consider SPARQL as another natural language and employ machine translation techniques for data access. Let’s now dive into a more technical explanation. Inspired by the Neural Programmer-Interpreter pattern introduced by Reed & De Freitas in 2015, the core of a Neural SPARQL Machine is composed of three modules: a Generator, a Learner, and an Interpreter. The aim of the Generator is to create the training set using the underlying knowledge graph and query templates. We define a query template as an alignment between a natural-language question and its respective SPARQL query, with entities replaced by placeholders, as in the following example.
Where is <A> located in?
SELECT ?x { <A> :location ?x }Pairs of translations are thus created by replacing <A> with a chosen amount of labels (entity names) on the left and URIs (entity IDs) on the right.
Where is London located in?
SELECT ?x { :London :location ?x }
Where is the Colosseum located in?
SELECT ?x { :Colosseum :location ?x }
Where is Mount Everest located in?
SELECT ?x { :Mount_Everest :location ?x }As SPARQL is code, it firstly needs to be transliterated into a sequence that resembles natural language. The training set is forwarded to the Learner, a deep neural network that translates a sequence of tokens in natural language into a sequence which encodes a SPARQL query. Our implementation features a recurrent neural network based on Bidirectional Long Short-Term Memories, however recent research showed that other architectures can perform even better³.
At prediction phase, the final structure of a SPARQL query is then reconstructed by the Interpreter using predefined heuristics. In natural language translation, the absence of a small part of speech such as a comma almost never changes the meaning of a sentence. Conversely, if the target language is code, the compiler or interpreter may complain of a syntax error. The goal of the Interpreter module is to fix these types of errors and return a valid query to the user.
The core architecture, as seen in the figure, looks promising. However, it is still very far from a fully autonomous system as long as query templates require human input. Top research teams have shown it is possible to infer question templates from data and discover new ones by mining large textual corpora on the Web⁴. To this end, the original architecture¹ was extended to envision a more autonomous and smarter Neural SPARQL Machine², seen below.
The points of strength of Neural SPARQL Machines are listed in the following. Given our latest findings², they can:
- Handle synonyms and equivalent expressions via global word embedding models, which are maps where similar words stand next to one another.
- Handle ambiguity via entity ranking, which means giving more importance to the entities that are more central in the data.
- Handle compositionality via curriculum learning, a practice where the model takes baby steps by learning to solve simpler problems first.
- Solve multiple problems at once, as most question answering systems rely on separate modules addressing named entity recognition, intent classification, and query composition.
- Easily support multiple languages by translating templates using machine translation and employing language-specific word embedding models.
- Be relatively fast and memory-efficient while translating questions into queries once the model has been trained.
- Be explainable by highlighting the matching parts of question & query using attention weights and reporting the interpretation of the user’s intent back to the user.
In the next articles, we will show how to query a Neural SPARQL Machines model and how to train a new model from scratch. The source code is available in this GitHub repository.
References
[1] Tommaso Soru, Edgard Marx, Diego Moussallem, Gustavo Publio, André Valdestilhas, Diego Esteves, Ciro Baron Neto, SPARQL as a Foreign Language. SEMANTiCS, 2017.
[2] Tommaso Soru, Edgard Marx, André Valdestilhas, Diego Esteves, Diego Moussallem, Gustavo Publio, Neural Machine Translation for Query Construction and Composition. ICML NAMPI Workshop, 2018.
[3] Xiaoyu Yin, Dagmar Gromann, Sebastian Rudolph, Neural Machine Translating from Natural Language to SPARQL. CoRR, 2019.
[4] Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, Gerhard Weikum, Automated Template Generation for Question Answering over Knowledge Graphs. WWW, 2017.
