Explanations from alleged masters of shape dissection

Andy Matuschak
Aug 28, 2018 · 2 min read

We’d just invited a few thousand students to try our latest experiment in online open-ended response activities — a controlled trial with pre- and post-test questions focused on concept transfer. We started reviewing their work, but the data from the control group stopped us in our tracks. We got page after page of responses like this:

Image for post
Image for post

This kind of answer wouldn’t be so surprising, except that all of these students had just solved four numerical “find the area of this polygon” problems perfectly, including one with exactly this shape.

We were testing an activity in which students would interact with each others’ open-ended explanations and ideas. They’d learn about finding the area of polygons by dissecting them into simpler shapes.

Our control group watched a video lecture on the subject, then worked through typical textbook-style exercises on the subject, like these:

Image for post
Image for post

Plenty of students solved every one of these exercises correctly. Typical standardized tests might suggest that these students had mastered shape dissection. But a third of those students couldn’t give even a partial answer to the “explain why” question above; another third gave poor or incomplete answers.

We didn’t design our experiment to look at this issue: we included these area-finding problems as a “dummy” activity to parallel the richer discussions in our experimental group. Maybe the issue is that the “explain why” prompt involves more algebraic knowledge. Besides, our experimental population is not necessarily representative.

But the intense gap between the procedural and conceptual assessments here still shook me.

Have you ever gotten an “A” in a class but felt you didn’t really understand the material? That seems to be exactly the disparity we’re seeing here. How many numeric-input or multiple-choice tests would exhibit the same problems?

We’ve long discussed our suspicions around scores taken from this kind of problem; we didn’t expect to confront such jarring examples so directly!


Originally published at klr.tumblr.com.

Khan Academy Early Product Development

Andy Matuschak

Written by

Wonder, blunder, salve, solve! Exploring empowering future possibilities in education with team at Khan Academy, where I lead Early Product Development.

Khan Academy Early Product Development

Working notes from explorations into future possibilities in education

Andy Matuschak

Written by

Wonder, blunder, salve, solve! Exploring empowering future possibilities in education with team at Khan Academy, where I lead Early Product Development.

Khan Academy Early Product Development

Working notes from explorations into future possibilities in education

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store