FQuAD: French Question Answering Dataset
To stimulate French AI: the first native French QA Dataset
Today, we introduce FQuAD, the first native French Question Answering Dataset. We finetuned the CamemBERT Language Model on the QA task with our dataset, and obtained 88% F1. To see it in action, check out our trained QA model !
Table of Contents
- Introduction
- Dataset Collection
- Dataset Analysis
- Experiments
- Test it yourself !
Introduction
Question Answering
As the name indicates, the task of Question Answering (QA) consists of finding an answer in a particular text given a specific question.
Recent advances in NLP have improved the state-of-the-art on this task, with the models being trained & evaluated on datasets such as SQuAD. However, these models being very much language dependent, and the datasets being almost solely available in English, French QA has not seen the same growth as its English counterpart.
Extension to French (and other languages)
English QA datasets are big (100,000+ questions). Current low-resource approaches to create datasets in other languages include translating the English ones to the desired language, translating text and question to English during the prediction, or using a multilingual language model. However, none of these options get quite the same results as native crowd-sourced language specific datasets.
Dataset Collection
Paragraph Collection
In this work, we chose the latter option, and created a crowd-sourced French QA Dataset of more than 25,000 questions. Following SQuAD’s approach, we randomly sampled 145 articles from Wikipedia’s French quality articles, further split into paragraphs.
Question & Answer Collection
With our own annotation platform, crowd-workers were asked to create 3–5 question-answer pairs per paragraph
This is where the current dataset stands (it’s being extended):
Additional answers collection
Human annotators may not all select the exact same text span when choosing the answer.
Quand fut couronné Napoléon ?
Possible selected answers in the text: mai 1804, en mai 1804, 1804
In order to decrease this annotation uncertainty, for the test set (dev set underway), we provided the crowd-workers with already created questions, and asked them to find the corresponding answer. This way, all test set samples have 3 answers from different annotators.
Some examples from the dataset
Dataset Analysis
We collected a diverse collection of questions and answers, which makes for robust QA models.
The crowd-workers were encouraged to ask difficult questions, and it shows in the dataset. Difficult implies linguistic variations between the question and its corresponding answer. This can mean using synonyms, changing the syntax, requiring world knowledge or information over multiple sentences to answer the question.
Experiments
With the dataset now constituted, the goal was to finetune a Language Model on the QA task. Thanks to the recently released French Language Models, CamemBERT and FlauBERT, the process of creating such models becomes straightforward with the HuggingFace library.
Results
We fine-tuned the Language Models on FQuAD, as well as a translated version of SQuAD and a combination of the two.
Our best model, CamemBERTQA, reaches a performance of 88.0% F1 and 77.9% EM on the FQuAD test dataset. Interestingly, we can see a strong bias between translated and native french data, reflected in the scores on the respective test sets depending on which training set was used.
Performance Analysis
If we look at how the model performs on the different question & answer types, we see that it works best on structured data likes numbers or proper nouns, but still manages to produce adequate results on more complex types.
Learning Curve
In order to better understand how the number of samples affected the model’s performance, we carried multiple runs with subsets of different training sizes, and evaluated on the same test set. As more samples are added, performance improves.
