How A/B Testing Powers Pedagogy on Coursera

Dan Saber
Dan Saber
Jun 14, 2018 · 2 min read

A/B testing is used throughout the tech world to build more engaging and valuable products. By randomizing users into either the current baseline experience (control) or a new experience (treatment), companies can measure the causal effect of the new experience on the outcomes they care most about.

Coursera’s platform enables instructors and researchers to bring that same technology and methodology to pedagogy. We do this by randomizing enrollees into different versions of the same course …

Image for post

… and then observing the effect of different teaching inputs (such as videos) on learning outputs (such as course completion).

Image for post

Our approach has proven powerful in a range of contexts:

  • It has informed content presentation formats. For example, instructors have used our infrastructure to test videos vs. slideshows vs. reading for optimal delivery of a given topic.
  • It has informed how we balance formative with summative assessments, in particular because formative assessments have been found to improve learning outcomes.
  • It has informed how we motivate and reward learners, and reminded us to pay careful attention to psychological factors when designing our platform. For example, instructors have found that having learners state a goal upfront improves course progression, especially among historically disadvantaged groups.

Coursera instructors have more than a dozen experiments live on the platform today, and they — and we — are closely monitoring the results, and adapting the platform and content based on what we learn. As one example, Chris Brooks, from the University of Michigan, wants to know: Do subtle gender cues impact female learners in STEM content? He randomized the appearance of male versus female workers in the background of some videos, and results are forthcoming.

By empowering our partner community to experiment, we hope to accelerate the pace of pedagogical innovation on the platform, bringing us ever closer to achieving our mission of transforming lives through the world’s best (and most thoroughly A/B tested!) education.

. . .

Interested in applying data science to education? Coursera is hiring!

Coursera Engineering

We're changing the way the world learns!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store