re: badges in education technology — what the psychologists found
Badges and better labelling can incentivise better practice in research — and could be a useful way to spur better decision-making in education.
In favour of this argument — check out this recent piece from FiveThirtyEight, highlighting the success seen by the Center for Open Science (COS) and the journal Psychological Science using badges to encourage researchers to become more transparent.
While the badges — rewarding the utilisation of publicly available data, publicly available methodology and for committing to a research design in advance of starting the project — may be technically “meaningless,” they work. The movement is seeing success, as noted by Brian Nosek (co-founder of the COS) due to its ability to tap into human nature and create conditions for people to voluntarily improve their behaviour.
As silly as they might seem, Nosek said the badges served a well-established purpose, by giving researchers a visible means to communicate information about their identities, beliefs, values and behaviors. People use such signaling all the time — think bumper stickers and hipster beards. Badges give scientists a way to signal that they care about research transparency, Nosek said.
Not to rip off FiveThirtyEight or academic journals, but research shows a clear and dramatic expansion in the reported use of open data, captured in these findings:
We demonstrate that badges are effective incentives that improve the openness, accessibility, and persistence of data and materials that underlie scientific research.
…. so what about education technology?
Researchers may not be for-profit companies, but human psychology holds true. This study reinforces the potential of badges for providing an important part of the answer of how to improve research in edtech.
We can learn three things:
Partner with an independent voice: COS is a non-profit organisation, dedicated to “increase openness, integrity, and reproducibility of scientific research.” Their work provides credibility. This strengthens their signals (badges) of good behaviour. Particularly in education (with competitive, commercial and political interests), companies would need an independent body to provide strong signals of good behaviour. Imagine if a respected institution like What Works Clearinghouse partnered with a group like Digital Promise or the Education Endowment Foundation (UK) and turn their guidelines for research into practical tools for fostering good behaviour in edtech research.
Focus on fostering good behaviour (not policing bad): Rather than focusing on “how do we catch bad actors?” focus on fostering good actors. Tap into what is known about human behaviour to create conditions that will reinforce better actions and mindsets. Leverage the work done by groups like the Behavioural Insights Team and create easy, attractive and timely ways for education companies to create a social network of good practice. When amplified across the expanding ecosystem of edtech companies, little actions could make for dramatic gains. COS saw the fruit of their seemingly small actions for academic research, education should follow.
- Transparency, transparency, transparency: With the exception of protecting commercial and proprietary information, education research should learn from COS and increasingly value transparency. It’s important for people to understand how data was collected, what methodology was utilised and how that data was reported. This remains complicated. Balefire Labs is taking admirable steps to provide product reviews of education products — offering feedback on instructional design and usability. But when it comes to research and “evidence of effectiveness research,” Balefire is forced to rely on linking to whatever research might be provided by the vendor. As of yet, there are no tools or easy ways to evaluate or understand that effectiveness research.
The challenge of how we conduct, report and understand education research remains in the headlines. As an example, take the Hechinger Report coverage last week of a new study from the Harvard Center for Education Policy Research on DreamboxLearning.
The headline read — Customized math lessons could help students learn more, research says. Digging into the article (and the actual research) made it clear that it was hard to know how much support there was for the headline. Reporting and research showed that variations in levels of usage and student types left researchers uncertain what learning gains could be attributed to Dreambox. If reporters and researchers aren’t certain how to interpret these results, how can (or should) a parent?
This is not a criticism of either Dreambox or the Hechinger Report. Efficacy studies are complex and hard work.At the end of the day, the complexity of education research can rarely fit in a headline. Dreambox and Harvard should be applauded for their commitment to the hard work of measuring impact. Greater transparency can only help improve the conducting and coverage of education research.
Finally, thanks for the feedback and insights. I particularly appreciate Rene and her thoughtful reply, highlighting the limitations of nutrition labels and challenges of trans-disciplinary research in answering this problem.
She rightly makes the point that, to make progress, there must be, “interaction/dialogue/feedback loops between all elements. In other words, the “special sciences” must collaborate, the stakeholders in education must collaborate, and it ends up looking like a beautiful, intricate web with many, many nodes and connections.”
That web might look like the Digital Promise Research Map or it might be something a bit more loose, dynamic and practical — closer to Wikipedia — a place where even this conversation can be a small part of that larger, more impactful tapestry.
