Measuring outcomes of high schoolers taking Computer Science courses

A case study on using Coursera in the classroom

In 2017, Coursera began working with the Bronx Science High School to provide students access to Computer Science (CS) courses over the summer, specifically the Fundamentals of Computing Specialization from Rice University. As part of the program we surveyed the students before and after program participation and measured the impact of the Coursera program on learning outcomes and on their interest in studying CS in college.

As a baseline, we asked students to describe their familiarity with and interest in CS during a pre-participation survey. The majority of students had some programming background already: more than half had taken at least one CS class, programmed for one or more years, used Python before, and written CS programs. Most students also expressed interest in taking additional CS classes in the future.

We followed up with an additional survey at the end of the Coursera summer program, and compared with the pre-program results to measure impact. Of students, 96% reported an increase in their understanding of CS, and 91% reported an increased desire to study CS in college.

The table below shows the average change from pre to post for each characteristic (measured on a 5-point scale) as well as whether that change was significant under the corrected significance value (using α=0.05). Significance is based on a Wilcoxon rank test to adjust for our paired data, and with the Bonferroni correction to correct for our multiple hypothesis testing. We see that the impacts of the program on students’ confidence and interest in the field were generally positive: self-reported understanding of CS, programming ability, knowledge of their favorite programming language, and ability to build programs and apps all rose significantly.

Finally, in both the pre and post surveys we asked students to rank the salaries of six different careers: computer scientist, lawyer, doctor, teacher, musician, and engineer. According to national averages from Glassdoor, students should have placed computer scientist second, after doctor and ahead of lawyer.

In the pre survey, only 7% of students correctly put computer scientist second (and none put it first); instead, 93% ranked computer scientist too low — generally either third or fourth, often behind both lawyer and doctor and sometimes behind engineer as well. (Eighty-two percent correctly placed doctor first.) By the end of the program, in contrast, students’ perceptions of the market value of a computer scientist moved upward toward actual: A Wilcoxon rank sum test for the change in salary rank suggests that the average change of -0.3 was significantly different from zero with a p-value of 0.04. This negative rank means that the same student on average ranked the relative salary of computer scientists higher after the program than before; that is, by the end of the program students perceived increased career prospects for computer scientists relative to the other careers listed.

All in, while the Bronx students were relatively experienced in CS prior to the Coursera program, the incremental training via Coursera meaningfully increased self-reported interest in and understanding of CS, and an improved (and more accurate) perception of the career prospects for computer scientists. Free-form comments from students reflect similar sentiments; as one Bronx learner wrote, “Taking this Coursera program has given me important and deeper insights on computer science. I am way more motivated to pursue a career in Comp Sci.”

The results from this study suggest that serving Coursera courses to students in a blended learning model can have strongly positive impacts on their interest and understanding of the field, even among students already familiar with the topic area. As we at Coursera think about ways to engage students and introduce them to different career opportunities, using courses in high school classrooms is one model to drive positive outcomes for learners.

Interested in applying data science to education? Coursera is hiring!