IJCAI 2019: Towards 5k submissions
This year IJCAI was HUGE: 4752 submissions, 850 accepted papers.
But how much is it, 5k submissions?
If a standard sheet of laser-printer paper weighs about 5 grams and on average there are 7 pages per paper, then 5k papers weigh about 175 kilos. Nah, it’s not as huge as this Elephant in the Room, but it is around 2 baby elephants.
Or if we consider the length of an A4 sheet (210x297 mm) and then put each sheet of paper on the ground one after another, we will cover more than 10 kilometers. In Macao, where the conference took place, we could build a Submission Road from the very south to the very north of the city.
Now it should be easier to visualize the load of submissions the reviewers went through :)
There were about thirteen parallel tracks, usually with a big part with core ML and only one-two NLP-related. I was surprised to see the great popularity of the following topics:
- ML: Statistical Learning, Tractable Deep Learning — the main goal of which is boosting explainability
- Knowledge Representation and Reasoning (KRR): Logical Formalisms + Computational Learning
And it’s all for a reason. How many times you simply didn’t have enough data to make use of a Neural Net? Let’s look at the results of the classification model Logistic Circuits. While not being a Neural Net, it is rather an example of combining ML with KRR. It was presented by invited speaker Guy Van den Broeck, an Assistant Professor from UCLA, directing the Statistical and Relational Artificial Intelligence lab. In the tables below two main advantages are demonstrated: first, the Accuracy table shows that this model is comparable with Neural Nets and second, the Number of Parameters table shows that it is significantly smaller in size.
SemDeep workshop
It was my pleasure to attend the Semantic Deep Learning workshop for the second time! Last year it was at ISWC in Monterey, CA. Dagmar Gromann, a University of Vienna Assistant Professor, is a SemDeep committee member. She certainly has a gift of making things worthy of attention and a front row.
The best paper went to Extending Neural Question Answering with Linguistic Input Features. The work was made in the frame of the Prêt-à-LLOD research project, which I am as well a part of 😉 Check it out, we promote the usage of freely available linguistic data!
For further reading on the workshop, the summary is well presented in Artem Revenko’s post.
SCAI workshop and Deep Pavlov library
DeepPavlov is an open-source library with up-to-date QA models, pre-trained embeddings and all the good stuff under Apache 2.0 license. It can help you in NLP research and the development of production-ready chat-bots or other complex conversational systems.
Its founder Mikhail Burtsev is as well:
- Head of Neural Nets and Deep Learning Lab at Moscow Institute of Physics and Technology, a top national university in Russia
- Search-Oriented Conversational AI workshop (SCAI) organizer at IJCAI 2019. This workshop summary you can find in Michael Galkin’s post
- Lead Organizer of Conversational Intelligence challenge (ConvAI) at NIPS 2017 and 2018
Check out their AI school schedule and current competitions at http://deephack.me/
Angry Birds AI Competition
Angry Birds has an infinite number of possible actions to choose from and the exact outcome of each action is unknown in advance. This makes the game very hard for computers to master. So hard that for the whole eight-year existence of this competition AI couldn’t beat humans. Interestingly, the best systems are rule-based using statistics and heuristics. Deep Learning models are very good at.. taking the last place 🤫 Next time your friend says that AI is about to take over the world, use this valuable argument.
Best Papers
Boosting for Comparison-Based Learning: a novel approach for comparison-based learning. It is a case when a user is presented with a new object and asked to reason about it, based on previous instances — for instance, object C is closer to A than to B.
Portioning Using Ordinal Preferences: Fairness and Efficiency: a new approach around divisible portioning. It is when people can vote on which area a particular resource should be spent, e.g. money budget or time.
Out of Sight But Not Out of Mind: An Answer Set Programming Based Online Abduction Framework for Visual Sensemaking in Autonomous Driving: tracking and explaining the scenes, for example, a cyclist got hidden behind a car in front. It is some form of question answering over dynamic visual imaginary.
Following my 2019 summer tradition, here are 6 selected papers by me on Question Answering and Semantics. Enjoy!
Question Answering
1. AmazonQA: A Review-Based Question Answering Task
CMU students introduce a new publicly available Amazon QA dataset consisting of 923k questions, 3.6M answers and 14M reviews across 156k products.
- the answers are provided by users in a real-world scenario
- the questions are frequently only partially answerable based on the reviews, leaving the challenge to provide the best answer
- the dataset is large, comprising several categories and domains thus possibly useful for learning to answer out-of-domain questions
It is built upon the dataset of McAuley and Yang, who scraped the product pages of amazon.com from May 1996 to July 2014, spanning 17 categories of products.
Authors propose as well a method that combines information retrieval techniques for selecting relevant reviews (given a question) and reading comprehension models for synthesizing an answer (given a question and review).
To demonstrate the challenging nature of this new task, the evaluation of numerous models for answer generation and strong baselines are proposed in the paper.
2. Knowledge Base Question Answering with Topic Units
Lan et al. address two following limitations of KBQA:
- oftentimes an entity mention in a question is ambiguous and an entity linking tool may not link it to the correct entity in the KB
- entity linking step cannot be trained using the final KBQA results
and propose to replace the standard topic entity linking module with a novel topic unit generation-and-scoring module.
In their approach topic units include linking not only to named entities but also to other KB units such as entities containing common nouns and relation type, as described in the example below:
The point I’m missing in the paper is the situation when we don’t have all the topics in the KB — do we generate them on-the-fly in that case?
3. Neural Program Induction for KBQA Without Gold Programs or Query Annotations
Neural Program Induction (NPI) is a paradigm for decomposing high-level tasks such as KBQA into executable programs by employing neural models.
Typically, this involves two key phases:
- Inferring input program variables from the high-level task description
- Generating the correct program sequence involving these variables
This raises the following challenges:
- Noisy query annotation in the absence of any supervision can lead to catastrophic forgetting while learning
- The reward becomes extremely sparse owing to the noise
To deal with these challenges Ansari et al. propose a noise-resilient NPI model, Stable Sparse Reward-based Programmer (SSRP). Two notable distinctions from other existing NPI models are that it learns program induction in absence of supervision from gold programs; and for both during training and evaluation, it has to handle noise in the query annotation, so severe that it renders up to 90% of the questions unanswerable.
On complex KBQA datasets, SSRP performs at par with hand-crafted rule-based models, and in the noisy settings outperforms state-of-the-art models even with a noisier query annotator.
Semantics:
1. Knowledge Aware Semantic Concept Expansion for Image-Text Matching
Image-text matching is a vital cross-modality task in AI. The majority of existing models simply detect semantic concepts from a given image, which are less likely to deal with long-tail and occlusion concepts.
Frequently cooccurred concepts in the same scene, e.g. bedroom and bed, can provide common-sense knowledge to discover other semantic-related concepts. Shi et al. in a mutual work of Beijing Institute of Technology and UCLA develop a Scene Concept Graph (SCG) by aggregating image scene graphs and extracting frequently co-occurred concept pairs as scene common-sense knowledge.
Specifically, semantic concepts are detected from images and then expanded by the SCG. After learning to select relevant contextual concepts, their representations are fused with the image embedding feature to feed into the matching module.
To extract semantic concepts following steps are performed:
1. Concept Detection Module extracts accurate concepts on a small vocabulary.
2. Use the assembled Scene concept Graph to expand more contextual (read as undetected) concepts by Concept Expansion Module. However, the expanded semantic concepts contain a lot of noise.
3. Concept Prediction Module predicts reliable concepts that are prone to efficiently enhance the image representation based on the visual feature. After the prediction, these filtered semantic concepts are fused with the vision feature and generate an enhanced representation of an image.
4. Finally, cosine distance between enhanced image feature and text feature, extracted by Sentence Encoder, is used as the similarity score.
What I would like to know more about is the prediction module. In the paper, the image-fusion process is described very shortly. Why, for instance, a laptop’s monitor doesn’t get a high score? Is the model flexible enough so with turning the laptop upside down, it will still predict “A tabby cat is lying on a monitor”?
2. Unsupervised Embedding Enhancements of Knowledge Graphs using Textual Associations
Knowledge graph embeddings are instrumental in representing and learning from multi-relational data. Such precisely structured data is usually limited in quantity and scope. Therefore, to fully optimize the embeddings it is important to also consider more widely available sources of information such as text.
Veira et al. describe an unsupervised approach to incorporate textual information by augmenting entity embeddings with embeddings of associated words. The approach allows it to be integrated with existing embedding models.
Two distinct forms of textual data are considered, with different embedding enhancements proposed for each case. In the first case, each entity has an associated text document that describes it. In the second case, a text document is not available, and instead, entities occur as words or phrases in an unstructured corpus of text fragments.
Experiments show that both methods can offer an improvement in the link prediction task when applied to many different knowledge graph embedding models.
3. Relation Extraction Using Supervision from Topic Knowledge of Relation Labels
In this paper Jiang et al. mine the topic knowledge of a relation to explicitly represent the semantics of this relation. They model relation extraction as a matching problem so that the matching score between a sentence and a candidate relation is predicted for an entity pair.
An example of describing the effectiveness of the supervision provided by topic knowledge:
A novel deep matching network is proposed to detect the relations of entity pairs based on topic knowledge. This framework is suitable for many classification-based NLP tasks.
Conclusion
As my background is more on the side of dealing with the unstructured texts, after attending IJCAI 2019 and ACL 2019, I’m still amazed at how popular are knowledge graphs in NLP and specifically in Question Answering domain. Curious to see, how it will go on in the future.
Some conference-related details
- The conference was held in Macao. As the humidity was over 80%, I couldn’t stay outside for more than 5 minutes 🥵
- The venue was the Venetian Macao, so far the largest casino in the world and the largest single structure hotel building in Asia. My takeaway is that any hotel with a closeby loud casino is not the best place for a conference.
- Top 3 accepted papers by country: China: 327, USA: 169, Australia: 37.
