by Michael Marder, Executive Director, UTeach Austin
The largest exam rollout in the history of the College Board took place in 2016–2017 with the new AP CS Principles exam. Several organizations that provide CS curriculum and teacher support are tightly connected with university research groups and have had National Science Foundation funding support. However, the single biggest player by far is a nonprofit organization with strong Silicon Valley connections, Code.org. It is interesting to note what happened when the test score results came in.
UTeach at the University of Texas at Austin is one of the university-based CS providers. Students who took the UTeach CS Principles course did 9% better on average than the rest of the students across the nation. Female students did 14% better when they took the UTeach course, black students did 13% better, Hispanic students did 13% better, and Native American/Hawaiian Islanders did 14% better.
UTeach announced these results on its website and they were subsequently shared throughout the CS Education community. The result was a spirited Facebook discussion of the role of test results in evaluating CS curriculum and professional development providers. Hadi Partovi, the founder of Code.org, argued in two Facebook posts and a blog entry that it was “dangerous” for districts, schools, and teachers to consider AP exam scores when selecting a program to follow. He suggested that the difference in exam results might mainly be due to Code.org’s emphasis on working with underrepresented groups
These points deserve an answer. Exam scores are not the sole measure of quality instructional materials and teacher support, but they should not be ignored. It does make sense for districts, schools, and teachers to pay attention to test scores when preparing students to pass a test. One of the primary reasons schools and students signed up for a CS Principles course was because AP courses offer tests that can lead to college credit.
If schools and teachers follow Code.org advice “not to factor in nationally reported exam results,” their alternative is to decide based largely on marketing and publicity. That requires money, and a lot of it. And when it comes to money, there is one provider with more than any other, raising $10 million to $20 million a year from philanthropy: Code.org.
Let’s back up and look at the landscape of the AP CS Principles course. The point is to broaden high school student participation in computer science. The College Board provides detailed descriptions of its courses, provides pacing guides, and administers the exam, but it does not provide specific curriculum materials and support. That is left to other providers. For 2016–2017, there were really two groups of providers: Code.org and everyone else. Out of roughly 2,500 schools that offered the course, 37% used the Code.org curriculum and 19% of the teachers were prepared by Code.org.
To have moved so quickly and operated on such a large scale is a genuine achievement, and Code.org has every reason to be proud.
But one should not dismiss out of hand the smaller, university-grounded, research-based efforts supported by the National Science Foundation. UTeach CS Principles is one of them; others include the Beauty and Joy of Computing, Mobile CSP, Exploring CS, and Bootstrap. Getting into the NSF club takes months of planning and writing and surviving a rigorous evaluation process. Then the NSF requires a very careful process of course development and inspection of results. None of the NSF-supported projects is extremely large. UTeach prepared around 200 teachers in 2016–2017 with both face-to-face and online sessions, and of them, around 100 ended up teaching the course. NSF funds made it possible to offer a considerable amount of support to teachers during the year. The UTeach work in 2016–2017 built upon five years of prior effort on a course called Thriving in our Digital World, led by University of Texas at Austin CS professor Calvin Lin, and developed largely by computer science educator Bradley Beth.
There are logical reasons that students did well in the first administration of UTeach CS Principles. We had been working on the course for five years, we had generous support from the National Science Foundation, the number of teachers we worked with was fairly small, and we offered intensive professional learning along with ongoing implementation support to teachers throughout the year. It would have been surprising if Code.org did as well when they ramped up to 10 times our size in one year without comparable experience or teacher support.
It should also be emphasized that the UTeach course is project-based, which is an instructional philosophy about as far away from explicit exam preparation as one can adopt, and was designed from the outset with equity in mind to help underrepresented students succeed in computer science.
Code.org is great at going big fast, and they have contributed tremendously to raising awareness about the need to improve CS education and the development of national CS education policy. But the NSF providers have a tremendous contribution to make when it comes to curriculum, professional development, and teacher support research, expertise, and know-how. We are good at getting high-quality results through working methodically and growing slowly.
Teachers prepared by Code.org had pass rates that were eight percentage points below the national average. Code.org responded by raising doubt about using test scores in general. They also raise doubts about whether UTeach obtained its strong results by working with a privileged population. This matter is settled by reporting pass rates for underrepresented groups, which UTeach has done and Code.org has not. Further research never hurts, but we already know that UTeach got especially favorable test score results with African American, Hispanic, and female students. It is extremely unlikely that a much more sophisticated analysis involving prior test scores and a variety of demographic and school characteristics in hierarchical regression models will give a very different result. An important demographic variable missing from the College Board reports is poverty, but it has repeatedly been demonstrated that race and poverty are so tightly connected that knowing race is, unfortunately, a very good proxy for poverty. The following plot showing poverty and minority concentration for every California high school illustrates this.
All the NSF-funded projects that have prioritized providing computer science education to disadvantaged groups have been working on this for years, and UTeach results from the first AP exam show success. NSF deserves credit for its sustained attention to quality, and this matters particularly now that NSF is experiencing very public criticism.
Code.org, with its Silicon Valley roots, is a real powerhouse. They could use their strength to collaborate with other groups to improve computer science education, to learn from what others know about curriculum and teacher support, and to help where others are weak, which is in sustaining programs and growing to national scale. If we could figure out how better to collaborate, imagine the impact we could have to provide the highest quality CS education for all students.