Instrumental Variables & Randomized Encouragement Trials: Driving Engagement of Learners

This is Part II of our Causal Impact @ Coursera series. (Part I is here)

Coauthored with Alan Hickey, Data Scientist at Coursera

At Coursera we use data to power strategic decision making, leveraging a variety of causal inference techniques to inform our product and business roadmaps. In this causal inference series, we will show how we utilize the following techniques to understand the stories in our data:

(1) controlled regression

(2) instrumental variables

(3) regression discontinuity

(4) difference in difference

This second post covers an application of instrumental variables in a randomized encouragement trial as a way to drive learner engagement.

Data plays an important role in helping us understand how people learn, enabling us to design a better in-course experience that is fun and engaging. A central question here is whether a particular learning style can lead a learner to be more engaged and thus more likely to ultimately complete a course. Specifically, we could hypothesize that bingeing on course content (completing a sizeable chunk of a course in one sitting) is better than splitting consumption over many learning sessions because the learner might forget important concepts or may not even return to the course after getting busy with other commitments.

So, let’s take a look: Does bingeing increase the likelihood of completing the next week in a course?

We define bingeing behavior as completing and starting consecutive weeks of a course within one day, looking at the relationship between this bingeing behavior and completion of the next week, adding in controls to test the robustness of our estimated effect, following the technique of controlled regression that we discussed in a previous post. The results of these two regressions are in the table below.

We see that there is a positive and significant relationship between bingeing and completing the following week, but because the coefficient changes significantly upon the inclusion of controls, we cannot say this is a causal relationship. Specifically, this positive correlation could just be self-selection by learners who are both inherently more likely to complete as well as more likely to binge because of higher motivation.

To rigorously test this, we decided to design an experiment where we randomly encouraged half of the learners in a selected sample to “binge.” Learners in our treatment group received a message immediately after completing a week of material that encouraged them to start the next week right away.

This experiment was designed as a randomized encouragement trial because directly stratifying on bingeing behavior is not possible. Instead we used the randomly assigned message to encourage learners to binge, meaning that receiving the message should be correlated with bingeing behavior but uncorrelated with everything else. Furthermore, this message will affect completion rates of the next week only through its effect on bingeing behavior. These two characteristics make it a great instrumental variable to test the causal effect of bingeing behavior on week completion.

We can therefore measure the causal effect of bingeing on week completion from this experiment using two-stage stage least squares, with an indicator of the message receipt as an instrumental variable for bingeing behavior.

For a primer on instrumental variables, randomized encouragement trials, and two-stage stage least squares, see this lecture from A Crash Course in Causality on Coursera.

The table below shows the regression output from this design:

• In the first column we regress whether a learner completed the following week or not on whether the learner exhibited bingeing behavior. This OLS regression shows a significant and positive relationship between bingeing behavior and week completion, but we can’t take it as causal because of the issues discussed above.
• The second column shows the results of the first stage regression where we regress bingeing on our instrumental variable of receiving the in-course message. We see a high F statistic above 14, indicating a strong first stage and a positive coefficient as our message was designed to encourage bingeing behavior.
• The last column shows the second stage regression results where we regress completing the next week on the fitted values from our first stage regression. Note that the coefficient on bingeing here is positive but not significant, suggesting no strong causal relationship between bingeing and next week completion.

Because the IV regression results showed no significant causal impact of bingeing behavior on next week completion, we concluded that we should not encourage learners to binge. Instead we should encourage them to follow the learning style that works best for them and their schedule, which is why we have focused on building personalized learning schedules, which we will detail in a future post.

Interested in Data Science @ Coursera? Check out available roles here.