Live Blog: Delft Data Science — Early Interventions to support online learners

This is a quick live blog from the session at the Delft Data Science Conference. I have tried to quote the presenters as much of possible, but due to the speed, I’ll probably have failed at some parts. Errors are mine, not the speakers! I’ll be refining this article during conference, if you spot errors and typos, please let me know.

Sessions was openend by Claudia Hauff and Christoph Lofi from the Lambda Lab. (You can find their slides here)

First presentation: Justin Reich from MIT

Justin Reich started of by stating: the initial frame of mind can really shape the overall experience of people.
If you start off on a good foot, you feel successful, that helps, you do better, that feels good. This is a positive recursive proces.
Of course you also have negative recursive processes. You get into the class, get negative feedback, become discouraged etc.

So therefore Justin is interested in how people start things.

Nudging

Justin’s research is closely related to Nudging.
An example of nudging can be found in organ donations.
When it is opt-in not a lot of people sign-up. When it is opt-out, not a lot of people sign-out.

It seems people just think: “I don’t care what you do with my organs, just don’t make check another box!”

The gap between intention and reaching your goal

Justin then introduced the Intention-Goal gap.
Early on, lots and lots of people start, then people fall out

Now this can be seen not only in education. This charts can be plotted everywhere, from exercise to the entertainment industry. It seems to be an attribute of the human condition.

But: the drop-off rates are not the same for everyone.
People from less developed countries, have lower completion rates (controlled for all kinds of thingsl like education level etc)

There are probably two main reasons:

• Physical barriers (electricity, internet access) This is something that we as data scientists can’t really change
• Psychological barriers. This we can target, so this is where we focus on.

So their goal is: jump in early on, do interventions and get them into positive recursive processes

Interventions

First type of intervention: planning
Second: feeling of being outsider can trigger negative recursive processes.

1. Planning

They’re all about goals, intentions and gaps. In the Go vote example, people asked: where are you going to vote, how are you getting there? The research shows that it helps people to take action.
But these domains still simple. Not just a ‘vote once’. It’s now ‘complete a course over several weeks, filled with lots of activities’

2. Outsiderness
The first type of this is inclusion: It is important that people feel welcome, included. Now it is not clear what triggers people to feel like an outsider.
Justin and his team created different interventions, sometimes direct, sometimes stealth. (an example of stealth is: try to make them feel like they help someone else)
The second type is value affirmation (todo: write a bit more about this)

The intervention
The intervention is within the pre-course survey. The first thing they try to funnel people towards is ask people ‘tell us about yourself’. Before they saw this just as a place to collect information.
Now they see it as the first place to help students feel welcome in the course.

Most important question in the survey is: ‘why are you here?’
It turns out lots of people have no intention of finishing the course. Justin is interested in people who signal that they want to finish the course.

Justin also introduced a new metric here: people who are just browsing, deciding up staying because they liked it so much.

The pilot studies are Treatment/control studies. For now they were just randomised, in future they will be stratified TC studies, pairing people with people like them.

The text above the intervention states ‘planning is really good. People who plan do better than people who don’t plan.’

At the text input, a substantial number people don’t write it down. It turns out that only matters a little bit (more on that later). So it seems that being exposed to question matters.

Two outcomes we’re interested in:

• honours certificate (free, above 60–70 score)
• verified certificate ($100 fee, ID check)

Results: completion rates are 14% versus 18%. A 29% relative increase. Justin points out that these are really good effect sizes given the that the costs are basically zero. If each one helps 5–6%, and there are hundreds of them, you can imagine this can really impact people’s lives.

The second type of intervention on social inclusion was less talked about. You can read more about it here.

Note: as control group they used study tips, because there is quite a lot of evidence that they just don’t have any effect. So you can have people do a little bit of work and be pretty sure that it’s a good control.

Replication

Justin and his team are really focused on replication. So all the links point to their work hosted on the OSF, which is the open science foundation.

In US, social sciences are in crisis. Many studies have failed to replicate.Now do huge scale replications and find that findings don’t hold up. Reason: people do research on dataset, and then keep looking (XKCD cartoon)

Risks of MOOC research: this is extremely possible. All kinds of ways how to measure it. Six ways to measure Time on Task. So Justin and his team publish in advance all of our hypotheses and analyses. If we do something else, be suspicious.

What I found really cool is that they even publish all their analyses up front on Github, so you can really check what they are planning to do and whether they did it.

My take-away:

very interesting work and a notable intervention on planning. Let’s try to implement it in much more platforms (we’ll certainly try it out in www.ihatestatistics.com) to see if we can replicate these effects in different platforms and contexts.