How Quizlet Creates Trickier Questions
By Tingting Lin and Shane Mooney
Do you remember studying for a test, drilling yourself on the same flashcard or text over and over again? This is a good way to memorize material, especially if you space out your study over time and try to engage active recall, and Quizlet has powerful tools for that.
This type of memorization is important and foundational, but often it isn’t your ultimate learning goal. For example, if you’re studying biology, you could just be memorizing term-and-definitions.
But you probably need to be able to flexibly understand each concept and how it relates to other concepts, as demonstrated below.
What’s key to understanding is the ability to flexibly access what you’ve learned, even in new contexts, with fresh examples. We heard from interviews that students use practice problems or practice tests to ensure that they’ve “really” got it.
One feature we created to let you test your knowledge flexibly is Smart Grading, which grades written (typed) answers by the overall meaning rather than requiring that you match the exact wording.
To better prompt and test for flexible knowledge, we set out to create additional advanced questions about what you’re studying, to ensure you’ve really learned the underlying concept, and not just memorized a certain phrasing of it. This variability of practice has been shown to improve generalization and robustness of learning.
Advanced Questions
Let’s say you’re studying the concept photosynthesis, defined as:
The absorption of energy from sunlight by plants, used for reacting carbon dioxide and water to produce glucose
Would you be able to recognize the concept if you saw it reworded?
This question covers the same concepts as the original definition, but it’s worded differently. We mix things up just enough to test that you did not simply memorize the original sequence of words.
Going a step further, we can ask a very different question about the same concept. For example:
This kind of question requires developing flexible knowledge of the concept and possibly integrating that knowledge with other concepts outside of what you’re directly studying.
Approach
How does Quizlet go about creating these questions? Our learners help us!
Quizlet has over 4 billion unique terms created by users. Of these, we extract 400 million eligible questions and answers to be used as advanced questions, spanning across hundreds of subject domains.
We further cluster similar answers so that a concept can have as many candidate questions as possible, using a technique called Locality-Sensitive Hashing (LSH). For example, answers such as photosynthesis, role of photosynthesis, and site of photosynthesis can be grouped together to provide candidate questions for the concept photosynthesis. If you’re studying photosynthesis, there would be tens of thousands of questions available on Quizlet for you to learn this concept more flexibly.
With so many questions to choose from, how do we know which ones to show students?
To best serve students’ learning needs, we had subject matter experts (SMEs) grade a sample of questions. With the manual labels as training data, we developed a system to rank questions with four factors: grade level, subject domain, similarity and quality:
- Grade Level
We find appropriate questions for the students’ grade level. For example, a typical high school level definition for photosynthesis is easier to understand than a college level definition.
2. Subject Domain
We choose questions from the same subject domain (classified by Quizlet’s academic subject classification model) that the student is studying. The examples below show how questions for photosynthesis may vary in different subject domains.
3. Similarity
We find close questions through semantic and string similarity comparisons between the terms they are studying and the candidate questions. We try to find questions similar enough to be relevant, but not so similar that it’s too easy.
For example, suppose a student is studying photosynthesis and practices with a question:
After the plant’s stem and leaves break through the ground, what process provides the food for the plant to continue to grow and mature?
Below are the sample questions we would exclude:
Below are the sample questions we would want to show as extra practice:
4. Quality
We measure question popularity (how many students studied it recently) and format quality (is it in a clear and readable format) to ensure quality. We are open to questions in multiple-choice, fill-in-the-blank, question-and-answer, or term and definition formats.
Combining all the four ranking factors above, we select the best advanced questions for each existing term.
To serve advanced questions to students, we created a new option in Quizlet’s Learn study mode for students who want to develop a deep understanding of their material. In this option, Quizlet first ensures that students are familiar with the original content in the set, then eventually works up to advanced questions.
Results
We were encouraged to see that most Learn users chose this new option, indicating they wanted advanced questions. We heard great feedback from users like:
I would love to see more! This feature allows me to check if I fully grasped the terms and definitions of what I am studying
they throw you off just like the real test might
…shake things up in a fun and interesting way that makes me want to use them again
Between user interviews, feedback, and metrics results, we validated that many students had learning goals beyond memorization, and we were helping them achieve those.
Future Directions
The right difficulty: Since these advanced questions are more difficult, students are getting these questions wrong on average much more than for our other questions. We don’t want them to be so hard as to discourage some students, so we plan to present them as multiple-choice questions (MCQs), which make them a little easier. Also, MCQs test for recognition rather than recall of the right answer, helping us overcome the challenge of introducing others’ content while you study.
Aligned to the course: Students are focused on what they need to study for their course, and need to be aligned to what their teachers expect. We’re collecting more course information from students, and hoping to use this to generate the most relevant questions for the course.
Expanding the use cases: Our current algorithm works well for conceptual subjects like science and humanities. But there is more work to do to enable them for foreign language and math learning. We would also continue broadening our research scope when new use cases emerge, as Quizlet’s mission is to make every student truly unstoppable.
Conclusion
At Quizlet, we ultimately want to empower students to have the most impactful study experience as possible, inspired by learning science and proven techniques. We think that asking students advanced questions on the concepts that they are studying is a big step towards that, helping them move up Bloom’s taxonomy of knowledge from memorization to understanding and beyond. We were able to uniquely build this by leveraging the huge amount of studying and content on Quizlet today.
We have much more to do, and if this sounds interesting to you, come join us!
— — —
Acknowledgements: Thanks to Anna Khazenzon and Ling Cheng for all their research and contributions to this post. Thanks to the Quizlet Learn team who contributed to bringing this to life for our users.
