What does $600K in NSF research funding buy?

Mike Lee’s 2017 Gidget birthday cake

Eight years in to my career as a professor I finally wrote my first NSF final report (this would have come earlier, but taking two years of leave to do a startup led to several no cost extensions). Because science and technology has been under attack for quite a while now politically in the United States, this seemed like a great time to step back and consider what NSF funding actually buys America.

The project was funded out of the now canceled “Computing Education for the 21st Century” program at NSF. With my collaborators Margaret Burnett and Catherine Law, I wrote a proposal to investigate whether framing programming as a game would equitably engage learners in more productive learning than other approaches to learning to code. We were one of several teams to be funded, in the amount of $600,000 for three years of research.

Here’s what we did with that money:

• We designed, implemented, and deployed the Gidget game
• We designed, built, and deployed the Idea Garden into the Gidget game
• We designed and built the Idea Garden for JavaScript and the Cloud9 IDE
• We designed, built, and evaluated a Problem Solving Tutor (not yet published)
• We designed, built, and evaluated a Programming Language Tutor (not yet published)
• We studied the role of data representation on engagement
• We studied the effect of in­-game assessments on engagement
• We studied the effect of Gidget on attitudes toward learning to code
• We studied the role of the design principles incorporated into the game on learning
• We studied the learning gains in the game relative to two other learning paradigms
• We studied the role of the Idea Garden in engagement and learning
• We studied the role of self­-regulation in programming problem solving
• We studied the effect of self-­regulation instruction on programming problem solving
• We studied the effect of the problem solving tutor on problem solving productivity (not yet published)
• We conducted a pedagogical analysis of online coding tutorials (to appear)
• We held four summer camps in Oregon and Washington based on Gidget for high school students, reaching over 80 rural and female high school teens.
• We held four Gidget open house sessions during CS Education Week (2013­-2016)
• We held a one day workshop with 25 Native American teen girls.
• We taught an Upward Bound Web Design course to 11 diverse high school students.
• We disseminated the results to several Microsoft product teams building computing education products.
• We served on advisory boards to two NSF­-funded computing education teams
• We served on a panel on Women in Computing at the Educause conference.
• We served on a panel on Women in HCI at CHI.
• We held a workshop at CHI about gender-­inclusiveness issues in software.
• We disseminated results to code.org.
• We created an extensive Computing Education Research FAQ.
• We attended a Dagstuhl workshop on assessment in computing education.

Across all of these activities, we taught over 10,000 people how to code via Gidget, ranging from ages 13–80, half of them girls and women. We trained 4 post docs, 6 Ph.D. students, 2 masters students, 18 undergraduates and 6 high schoolers in how to do computer science and computing education research. We produced Gidget, an online game for learning to code, that will be available for at least the next decade as a public resource. We published about 20 research papers, contributing an evidence base for better teaching and learning of computing through online learning technologies.

Is all of this worth $600K? One way to judge this is to quantify how much all of this education cost. If we look just at the 36 students we mentored, the public spent an average of $16K per student to each them rigorous research skills. If we include the 10,000 players of Gidget, plus the future players of Gidget for the next decade growth, the public spent an average of $2 per player to teach them a bit about computer programming and potentially engage them in future learning. From this perspective, the grant was one big education subsidy, promoting the development of highly-skilled STEM workers.

Another way to judge this is to anticipate the future impact of this training. Many of the Gidget players will be more likely to pursue STEM education because of playing the game (according to our research), which may have a net impact on the growth of the economy. Of the 36 students we trained in research that have graduated, many are faculty, UX researchers, and software engineers, filling much needed jobs in industry. The downstream impact of all of this training may be to fill unmet needs in the economy, allowing it to grow more efficiently.

Of course, the other import way to assess the return on investment of the work is to predict the long-term impact of the knowledge we produced. We’ve already disseminated the work to code.org, which is reaching tens of millions of learners in high school through their curriculum. Our research essentially serves to ensure that the learning those students are already doing is more effective than it would have been otherwise. That ultimately means a better, smarter, more effective STEM workforce in the future, which ultimately impacts the growth and productivity of the U.S. economy.

Across the 243 million U.S. taxpayers, each contributed a median of about 1/10th of one penny for this research to happen. What’s the return of that fraction of a penny? Will every American be more than 1/10th of a penny richer in 20 years because of our work?

Clearly, this exercise in trying to model and predict the impact of science funding is hopelessly fraught with reductive ideas about science. It even plays into the framing that house Republicans have used to attack science, accepting the premise that NSF is an investment in America, as opposed to something more idealistic, such as the betterment and survival of humanity. But in reflecting on all of these activities and the actual impacts they’ve had on the world already, I find the sheer scale of potential impact to be compelling in its own right and well worth the price. I can’t wait to see in 10 years what these impacts might be!