Behind the Design: LSAT® Test Prep
The design process behind creating a platform to level the playing field for the Law School Admissions Test
Summary
Khan Academy’s test prep platform aims to level the playing field for standardized testing, ensuring all students can build their confidence and access the benefits of personalized practice without having to pay huge fees for test prep programs. We offer official content made in collaboration with the test maker, personalized recommendations, full-length practice tests, tips and strategies available to any students — all for free.
Here we’ll share some of our behind-the-scenes design process!
Team Leads
Context
Love ’em or hate ’em, standardized tests are a necessary requirement for educational and professional opportunities. In the U.S., there are at least 50 standardized tests for admission to college, postgraduate studies, and vocational schools. The test prep industry is worth $346 million in the U.S alone — globally it’s much higher.
Traditionally, students in higher socioeconomic brackets have had access to expensive test prep classes and books that lower income students can’t afford. This is borne out by test scores, and ultimately translates to inequity in opportunity [1][2].
This implies that students are being kept from reaching their full potential in test scores, at least in part because they aren’t able to access the same resources as peers from wealthier families.
Recognizing this disparity, Khan Academy and the CollegeBoard partnered to provide official prep for the SAT in 2014 — for free. We launched the Official SAT Practice product.
Even as an MVP, the usage and efficacy results were positive:
- 40% of SAT test takers used Khan Academy, which was 2x the number of students using commercial test prep
- 20 hours of Khan Academy’s Official SAT Practice is associated with ~115-point increase (2x average gain)
We were onto something! The Law School Admissions Council spoke to us — could we make it available for the LSAT too?
Realizing this opportunity also applied to hundreds of thousands of students aiming to be our nation’s future lawyers, we decided to take what we learned from SAT and make a product that would work well for the LSAT.
Design Problem
How might we level the playing field for the LSAT by providing the best, personalized way to prepare on our test prep platform, for free?
Research & Strategy
We researched the landscape of both students who study for standardized tests and the landscape of standardized tests themselves, including the LSAT. We found that across tests, the same few user archetypes emerged:
- Preparers: Students who prepare well in advance for their test and have a lot of time to study and reach their goals
- Reviewers: Students who are fairly confident in their ability to do well on the test, and look at test prep materials as a way to verify their knowledge
- Procrastinators: Students who are late to the game and feel a lot of time pressure to build their skills before test day (we all definitely know this one!)
User Focus
We decided to optimize our product for the “Preparers” use case, since we believed we could help them the most and they would see the most gains from using our product. We anticipated whatever we created would also work somewhat for the Reviewers and the Procrastinators as well.
Test Research
The team went out and actually took the full-length, 3-hour LSAT during an official administration of it! 🤓This helped us to build empathy for learners and understand the test we were designing for.
System Architecture
Because our product is meant to incentivize consistent practice over a period of time, we drew a lot of analogous inspiration from game design and fitness / training apps to design a system that would motivate a user to repeatedly engage to practice over time.
We consulted game designer Melanie Lam to help inform progress mechanics that became the product’s foundation. Every week, we playtested with users to get a sense of whether the system made sense to and motivated them.
Though most students used their phones more and preferred phones for on-the-go practice, most people took full practice tests exclusively on their computers. However, given that mobile has overtaken web as users’ primary if not only device, we wanted to ensure whatever system we created could be future-proofed to work across both web and native.
Longitudinal User Testing
Once we had a functional product and a direction that users reported as compelling to them, we conducted longitudinal diary studies with real test takers studying for the LSAT who used our product to help them study. (Basically, “longitudinal” just meant we followed the same users as they used our product over a period of time.)
Students studying for the LSAT would keep a diary of their experiences using our product, providing feedback and explaining how they were using the product with their studying. This helped us to get feedback closer to realistic conditions — and helped us gauge how motivating our mechanics actually were over a sustained period of time — rather than try to design based off single usability sessions.
“Final” design
Nothing is ever final of course, but here was where the design ended up! After a lot of testing and iteration on the system, here is how the system is designed to work. (*Note: Not everything in the final product has been implemented yet, but stay tuned!)
Introduction
Imagine you’re a student thinking about going to law school. You’re studying for the LSAT, and you’ve heard Khan Academy offers free prep.
Diagnostic
Once you’re ready, take a diagnostic to see where your starting point is on all the LSAT skills, either as a full test or through section-specific diagnostics if you’re short on time. This helps the system get a sense of your starting point for accurate practice recommendations.
Confidence
As part of the diagnostic, you can indicate how well you know each answer. This does two things: 1) it’s a great pedagogical moment for you to reflect on how much you actually know, and 2) it will eventually help the system identify whether you got a question right because you actually know it, versus getting a question right because you guessed. This provides more information about how well you know the material and helps the system better tailor recommendations to you.
Schedule & goals
When you’re ready and you know what real test date you’ll be aiming for, you can set a goal score for yourself and a practice schedule.
Now the system will know the delta between your current position, where you want to get to, and how much time you have before your real test — so it can prioritize your practice recommendations accordingly.
Check-Ins
Each scheduled practice day, you’ll get a reminder that you need to practice that day, and you get a status update on how “on track” you are to reach your goal score according to the pace you’re going.
Practice Recommender
At the crux of your experience is the practice recommender list, which prioritizes question types for you to practice based on how important they are for you (an aggregate calculation of how often that question type appears on the test, and how well you’ve done on that question type). The recommendations change based on your performance.
Practice questions
Each list item is a test skill that you can practice. The more you practice the different question types, the more familiar you’ll become with them. If you get stuck at any point, we offer videos, hints, tips, and strategies, to help you understand why you got any problems wrong or to approach problems you’re not sure about.
Timed Mini-Section
Typically, practice is untimed so that you can focus on learning the material. However, periodically, you’ll be challenged to practice under timed conditions (Timed Mini-Section), so you can get increasingly accustomed to the timed pressure of the real test. The length of a timed mini-section is about half of a full-length section, so around ~17 minutes for the LSAT.
For timed mini-sections, we enable you to continue to answer questions after time is up, so you can know the delta between your score under timed conditions versus untimed conditions.
Practice Test
At scheduled intervals, you take a full-length practice test to get simulate the real experience and your score as much as possible. (The tests are real forms that were created and previously administered by the test makers).
Keep practicing and aiming to hit your goal score!
Results & Learnings
At the time of this writing, the product was still in beta — but even before launch, we got a lot of positive preliminary feedback from our beta release that went out to 8,000 test takers. Here’s a tiny sampling:
“I paid for a [competitor] course. Through the 4 months I’ve been in the course my score has been steady, but the biggest jump I’ve had was 6 points due to Khan Academy. I am so much more motivated to work on this website than I am in my [competitor] course. I am in love with this and will continue to use this more than my actual course I paid for.” — Student
“This is an awesome tool that will be very useful for those who are not able to to afford prep books or expensive LSAT classes. I wish it was available sooner, but better late than never!” –Student
“It is free and motivating. I am no longer intimidated about the LSAT, and I am no longer worried about how I will earn over $1000 for a prep course.” — Student
In addition, even in our pre-releases, we found students were using Khan Academy at higher engagement levels than the rest of our product, and meeting / surpassing our MVAL (“Monthly Very Active Learners”) goals of re-engaging with Khan Academy for at least 120 minutes. In fact, our MVAL conversion rate for LSAT was about 4x higher than other areas of the site during the beta!
All in all, the product was a huge win for the company — it meets a critical need for learners (which supports our mission), drives deep usage (which supports our strategy), and drives revenue (which supports our sustainability while still enabling us to be a non-profit).
We’re excited to continue to improve it as we get more learner feedback!
Thanks
Special shout-out to everyone on the team who helped make this product a reality!
- The Law School Admissions Council, for their passion about making LSAT prep available for everyone, and providing us with official content to help aspiring law students everywhere;
- Anju Khetan, our fearless PM, for shepherding the entire process and leading with relentless positivity and pragmatism;
- Amy Skerry and Celia La, our incredible tech lead and eng manager, for leading the product’s development;
- Our excellent engineers Matt Dunn-Rankin, Jeff Yates, Nick Breen, and Carter Bastian, who dove into complex technical problems in order to make the product work;
- Dave Travis, our content manager who led the team that created videos and articles to help make LSAT content more approachable, as well as our amazing content creators Anh-Chi Fuhry, Rosie Friedland, and David Rheinstrom;
- May-Li Khoe, our VP of Design, for her irreplaceable guidance with design direction;
- Kitt Hirasaki, for all his work designing the original SAT product and helping to use those learnings to shape the design direction for this new test prep product;
- Melanie Lam, for her game design expertise and helping to drive us to a clear and understandable underlying game mechanic for learners
