How to turn a hackathon project into a game that millions of students love (and learn from)
We recently released a new study game on Quizlet, Gravity. This post is about how we came up with the idea on a hack night, iterated with tens of thousands of students and teachers, and then released the game to millions on the Quizlet website.
Coming up with the idea
Every quarter, the Quizlet team organizes a company-wide hackathon to explore new ideas. We brainstorm new learning concepts, build product on new platforms, and experiment the product potential of new technologies. Over the last year, we’ve come up with everything from Quizlet for Apple Watch to multiplayer, real-time learning games to pronunciation feedback tools.
A few hack nights ago, three of us set out to rethink and redesign one of Quizlet’s study games, Space Race. In this game, students compete to type in the answer as the words fly across the screen. As the levels go up, the words scroll faster and faster and you rack up more and more points with correct answers. It’s extremely addictive and competitive.
We saw two big opportunities improving our existing fun, effective and simple game:
1) Add more educational value (potentially as educational as regular, non-gamified studying on Quizlet).
2) Make the game more fun and engaging by exploring an “outer-space” theme for Space Race.
We had 24 hours to figure out how to do that and get a prototype ready for a live demo at 5pm on Friday. Here’s the prototype of a new game called “Gravity” that we came up with.
Designing a game that is fun and educational
Introducing Gravity
In Space Race, words moved from the left to the right of the screen and you typed them before they got to the other side. Flipping the axis to be vertical to imitate “gravity” was our first outer space inspiration (and new name of the game — “Gravity”). The words would scroll down the screen towards your planet in outer space. Type the answer in correctly before the word hits your planet!
As the levels went up so did the speed and the points (as they had on Space Race). But now the increased speed could be represented as an increase in the gravity level, forcing the words to fall towards the planet faster. Taking the theme one step further, we matched this increased gravity after level 3 with a new planet where the gravity was higher.
Reinforcing what needs work
Games are fun — that’s objectively true. But the main goal of any experience on Quizlet should be learning. And it’s even better when the two can go hand-in-hand. Looking at some of the non-gamified study experiences on Quizlet, a common theme emerged on what made them educational: repetition of words you get wrong. And so for our new game, we decided to do the same thing. Words come in batches of seven and if you get any words wrong, they’ll come back in the next batch. Copying the answer when you get it wrong helps you learn it for the next round.
We doubled down on this concept further by changing how the game ends. In Space Race, you racked up lives as you got correct answers and each wrong answer killed off a life. Run out of lives and the game ends. In Gravity, however, when a word you already got wrong once came back in the next level — it was an asteroid (playing more on our outer space theme). Don’t answer the asteroid correctly and game over! This way, your score only goes up if you can learn the words you’re playing the game with.
Getting from prototype to beta
After our hackathon demo, we casually brought our work into a few local classrooms to see what students and teachers thought on our bi-weekly classroom visits. It was pretty easy to see that the game even in its 24-hour form was going to be a big hit.
Based on their enthusiasm, we slotted Gravity to be released this fall. But before we released it, we wanted to iterate on our original concept with students and teachers before releasing it to everyone on Quizlet. After some engineering work to get our prototype solid enough for tens of thousands of students to bang away at, we got to organizing a user testing and iteration plan.
Asking 20,000+ students what they think
We lined up a diverse set of local schools to test out Gravity and for the first time, we also engaged a group of 5,000 teachers across the US and the world to test out the new game with their students. In the end over 20,000 students played our beta of Gravity and gave us feedback that informed our final iteration.
In Classroom Testing
In September, the team who built Gravity visited a half dozen schools in the San Francisco Bay Area. The whole team participates in user testing and feedback — engineers, designers, and product managers — so that we all can develop a sense of what works for students. And hopefully, those experiences of being in a real classroom environment are front of mind for all of us when we’re building new products back at the office.
We tried to visit a diverse set of age ranges when testing out this game in particular. What is “cool” “fun” or even engaging for an 11-year old and an 18-year is can be extremely different. But that whole age range is the core of students who use Quizlet. So we made sure there was a mix of Middle and High Schools in this set of schools.
Remote Testing with 20,000+ students
While class visits are a core part of feedback cycle with our users, they’re not perfect. The rush of having the Quizlet team who built the product you’re testing out means that the feedback we get from students we visit in person is always going to be skewed. Plus, we can only visit so many classrooms but we want to get feedback from lots and lots of students (and students not just in San Francisco) to really validate that the changes we plan to make are the right ones.
After sending out a call for beta testers to teachers, we had nearly 5,000 teachers sign up in 12 hours! We sent them instructions on how to test out the game and collected feedback from them and their (20,000+) students.
Refining the game mechanics
Between our classroom testing and our remote testing (as well as the helpful internal feedback we’d gotten from the Quizlet team), Gravity’s major pain-points and areas for improvement were clear.
Explaining the game flow better
The missed terms turning into “asteroids” was totally confusing to students. The core concept made sense once we’d explained it to them, but no one could ever explain how the game ended after playing it once. The connection between the first time they saw the word and when it came back as an asteroid wasn’t clear.
Our solution was to make all terms be asteroids visually whether new or missed terms coming back. But now the returning, deadly terms would be red asteroids. And we added a little warning banner to let you know they were coming.
MOARRR Planets
In our beta, you changed planets at level 3 to imitate a new planet with a new gravity level. The students’ excitement when their screens changed was palpable in every class we visited. And one of our number one requests was to add more of that.
The surprise and variety was clearly something that would keep students wanting to play this game more and therefore learn what they were typing in better.
Adjustable Speed
The ability to adjust the speed in Space Race to account for different typing abilities has been a long time user request and it was no different for Gravity. So we came up with 3 “difficulty” levels with different point scales that we hope will make Gravity accessible to even more students.
Learning the terms you don’t know
From our first class visits, students complained the tedious wait for a term they truly didn’t know to get to the bottom of the screen before they could find out the answer.
For our next revision, we added the ability to “skip” right to copying the answer by pressing the escape key.
The decision to replace Space Race
When we originally cooked up Gravity, we imagined it as a better version of our existing game, Space Race, both more educational and more fun. And our goal was to replace Space Race with Gravity. But we wanted to make sure that our users would feel the same way. So in our remote testing survey, we asked students, “How would you describe this game to another student?” And the response was unanimous — it was a better version of Space Race.
“better version of space race”
“A lot like space race, but more interesting.”
“like space race but more fun and the words are going down.”
“It is similar to Space Race, but on a different axis and with better graphics.”
“Really love the Gravity game! Much better than Space Race!”
While we’ll keep the option to return to Space Race around for a while to make sure we transition it out well, this data gave us a lot of confidence that users would appreciate the change and see it as an upgrade.
Retesting our final product
With all these changes done, we wanted to make sure that our beta testers (both those who’d already used Gravity and totally new users) saw all these changes as improvements over both Space Race and our first Gravity beta.
Collecting ideas for future iterations
We collected another 3,000 feedback responses on our next iteration of the game and saw very positive stats. 80% said that they had fun and 84% said that they learned while playing the game. That last 20% that didn’t have fun and 16% who didn’t learn from it often complained that typing made it difficult for them to get a lot out of the game. This could be because they were younger students who didn’t have typing skills, or because the content was too long to type in a short amount of time, or the fact that you need to type the word exactly in order to get points.
We’ll definitely keep those issues in mind when we do further iteration on this game and think about the new games we’ll develop next. You’ll never find something that makes 100% of people happy but we think it’s important to keep in mind how we can keep striving for that.
Keeping loading times fast on slow school networks
One other area we were concerned about was the performance of all these fun graphic assets we’d added over a slow school network. More planets and asteroids meant a lot more large images. We’d made optimizations on simulated slow connections in our office but can’t know for sure until we try it out in the wild.
While you’re choosing your game settings (which requires no asset loading), we start to load the large planet assets you’ll need for the game. This way, when you’re ready to play, all the planets you’ll need for the first few levels are already loaded and ready to go without a wait. 96.3% of users never even saw the loading indicator we’d added for slow networks.
Releasing Gravity to millions of students
We released Gravity to the general Quizlet public on Sunday, November 22nd at 4pm. And so far 1.4 million people have played Gravity as of December 4th at 10am at an average of 3 games per user. The graph below shows usage climbing (with a dip for the Thanksgiving holiday):
And of course the feedback poured in — 20,000 people submitted their feedback on the game and it tracked roughly with the positive feedback we’d been getting from our last round of beta-testing. We found a few more edge-case bugs to fix but overall it was a very smooth transition.
More new games on Quizlet?
What we took away from our beta-testing process was that there was a real appetite for new games on Quizlet from both students and teachers. The excitement of watching students in a classroom play Gravity together was infectious — but it also made us really think about how much more potential there is to play with that realtime environment. So stay tuned for updates!
Play Gravity now!
Enough about how we ended up here — try Gravity for yourself on one of these study sets:
French colors
Spanish colors
World Series Winners
100 most common SAT
Let us know if you have any more suggestions for future iterations or want to report a bug.
This post was also published on the Quizlet blog.
