How A/B Testing Powers Pedagogy on Coursera

A/B testing is used throughout the tech world to build more engaging and valuable products. By randomizing users into either the current baseline experience (control) or a new experience (treatment), companies can measure the causal effect of the new experience on the outcomes they care most about.

Coursera’s platform enables instructors and researchers to bring that same technology and methodology to pedagogy. We do this by randomizing enrollees into different versions of the same course …

… and then observing the effect of different teaching inputs (such as videos) on learning outputs (such as course completion).

Our approach has proven powerful in a range of contexts:

• It has informed content presentation formats. For example, instructors have used our infrastructure to test videos vs. slideshows vs. reading for optimal delivery of a given topic.
• It has informed how we balance formative with summative assessments, in particular because formative assessments have been found to improve learning outcomes.
• It has informed how we motivate and reward learners, and reminded us to pay careful attention to psychological factors when designing our platform. For example, instructors have found that having learners state a goal upfront improves course progression, especially among historically disadvantaged groups.

Coursera instructors have more than a dozen experiments live on the platform today, and they — and we — are closely monitoring the results, and adapting the platform and content based on what we learn. As one example, Chris Brooks, from the University of Michigan, wants to know: Do subtle gender cues impact female learners in STEM content? He randomized the appearance of male versus female workers in the background of some videos, and results are forthcoming.

By empowering our partner community to experiment, we hope to accelerate the pace of pedagogical innovation on the platform, bringing us ever closer to achieving our mission of transforming lives through the world’s best (and most thoroughly A/B tested!) education.

. . .

Interested in applying data science to education? Coursera is hiring!