Ideas from the Code and Cognition Lab (2020 edition)
By Amy J. Ko and her students
Over my 20 year career as a researcher, it’s become increasingly clear how the substance of my work with students and collaborators is not our papers, but our ideas. And yet, most of how our academic communities disseminate discoveries is through papers: we tweet new publications, we give a conference presentation about them, we write blog posts about them, and occasionally, a journalist will report on a paper’s discovery. But throughout, the ideas themselves lurk in the background, hidden from view. Only in genres like promotion packets, invited talks, keynotes, do we ever get a clear glimpse into the ideas that are currently driving a scholar’s work. And too often, these are not public, archived, or broadly shared.
This has been bugging me lately, so my lab and I decided to try a new practice: each Spring, at the end of our academic year, we’ll collaboratively summarize some of the major ideas our lab has been working on. Some of these ideas have reached a level of maturity after many years. Others we have only just started publishing about, and will continue to for some time. And others still we have only just started thinking about, shaping our work to come. We’ll talk about our lab’s ideas in that order, giving a chronological sense of our collective trajectory.
Idea #1: Learning to read code is a crucial foundation to writing it
Several years ago, starting with work by my doctoral student Michael Lee in 2009, we began exploring what it means to know a programming language, and how to help people develop this knowledge. Mike and I very much started this work as designers, and we began enamored with the idea of starting programming language learning with reading and debugging activities instead of writing activities, in contrast to most dominant paradigms of CS education. We raised some money, built an educational game called Gidget that organized programming language learning around debugging (Lee et al. 2014), and found all kinds of interesting design factors that affected learners to continue to learning in the absence of a teacher such as framing the personality of the machine (Lee & Ko, 2011) and personifying data (Lee & Ko, 2012). While it wasn’t clear in the publications, it was clear in the design of the game and our observations of people playing it: engaging with a programming language’s semantics by watching programs written in a language execute, step by step, was a powerful instructional method.
While these design discoveries were incredibly fun, they focused more on engagement than they did learning. When Ph.D. students Greg Nelson and Benji Xie joined my lab shortly after, we started paying more attention to learning. Together, we found that really intentional direct instruction about a programming language’s semantics, rather than just immersion in semantics through debugging activities, could be quite effective at building robust knowledge of a language. Greg manifested this as a tutor in his PL tutor work (Nelson et al., 2017), Benji experimentally showed that sequencing learning to read code before writing it promoted more robust learning (Xie et al., 2018). Greg continued this work, contributing a method of assessing PL knowledge that built upon modern notions of assessment validity (Nelson et al., 2019), pointing to ways to integrate formative assessments into a tutor. And Benji synthesized these ideas into a broader instructional theory of why teaching learners to read semantics before writing them is necessary, with evidence that it was also effective (Xie et al., 2019).
Throughout, however, as we engaged many populations in programming language learning, we found that while learning was more effective when reading came first, this was often at the expense of engagement. Students viewed learning to read as a chore, disconnected from the possibility of creating with code. This surfaced a fundamental tension for teachers: does one start learners with reading to build a solid foundation upon which to write programs, but bore them, or engage learners in writing immediately, but face the punishing burden of debugging and semantics confusion? The obvious next steps for this work are to explore how to make learning to read in a programming language more meaningful and engaging, perhaps blending our work on Gidget with our more recent work on PL learning. Benji explored this in an upcoming Learning @ Scale paper, which explores the potential of learner agency to balance engagement and learning (Xie et al., 2020).
Idea #2: Metacognition and strategy selection mediate productive programming
Whereas our work on programming language learning has focused the challenges of learning syntax and semantics, our work on programming has focused much more on the process of programming. There were early glimpses of this work in my 2004 study on learning barriers in programming (Ko et al. 2004), which surfaced a whole range of strategic and process knowledge that learners required to create with code. But I didn’t have the words for it then. It wasn’t until Dastyni Loksa joined my lab in 2013 that we really started looking at problem solving, process, strategy, and self-regulation in earnest.
We started broadly and from an interventionist stance. Dastyni had the idea that by simply telling learners that programming required explicit thinking about process and careful selection of strategies that learners would be more productive in their problem solving. And after we ran a controlled experiment one summer, it turned out to be true: developing learners’ awareness of their process substantially changed their behavior, resourcefulness, and programming self-efficacy (Loksa et al., 2016). This replicated a long line of studies in other disciplines investigating the role of self-regulation and metacognition in learning, but brought it to the domain of computer science. From there, we went deep, investigating the granular types of self-regulation and metacognition that learners have (or more often don’t have), finding that these behaviors are infrequent, shallow, and ineffective, but that they do slowly develop over time without any specific teaching intervention (Loksa & Ko, 2016, Loksa et al., 2020). Dastyni used this observations to think about how to scale interventions, leading to his culminating dissertation work on a novel tutor that demonstrates an expert programmer exhibiting effective self-regulation skills, helping learners model that behavior. Early results show that this social learning approach may promote improved self-regulation skills without needing the kind of tutoring in our first study.
While Dastyni explored process, self-regulation, and metacognition, in parallel, we explored the role of strategies and strategy selection with my collaborator Thomas LaToza. Independently, he had the idea that perhaps much of programming expertise was actually strategic knowledge that could be encoded and shared explicitly. We wrote an NSF proposal together to investigate this, and have since been exploring many aspects of programming strategies. One initial look into this by my student Benji Xie was a strategy we taught for how to trace a program’s execution (Xie et al., 2018). It turns out that a really brief training greatly improved performance on tracing problems. Another study, which I led last summer, explored what happened if we taught explicit programming strategies to adolescents, finding that they’re quite effective at structuring learners’ programming work and improving self-efficacy, but only if learners can muster the impulse control to follow them (Ko et al., 2019). In parallel, we explored the same ideas with developers of varying experience, finding similar trends: if developers choose to use them, novices can perform at the same levels of experts, but many refuse to use them because of the freedom they lose (LaToza et al., 2020).
Our work on programming from a self-regulation and strategic perspective continues. Dastyni is graduating soon and will continue investigating self-regulation. We have a whole series of unpublished works on explicit strategies that explore strategy authoring, strategy sharing, and strategy selection. We’re very excited about the prospect of explicit strategies as a form of knowledge sharing that could transform how people learn to not only write programs, but design them, test them, debug them, and evolve them over time.
#3 Envisioning more critical CS education
The third major thread in my lab loosely concerns the role of critical perspectives on computing and how they relate to CS education. By critical, I mean observations about computing’s limitations and perils, both to individuals and society. While my lab’s work on programming language learning and problem solving has been slowly maturing, this thread is only just starting. But in a way, it’s been buried in the back of my head ever since starting my Ph.D. in HCI in 2002. Even back then, I had an unnamed skepticism about computing, always aware of its limitations, and always questioning its value. I think it came from spending a childhood with my brother, who has always seen technology as something fragile, shallow, and trite. I was always more optimistic about it, but with reservations. And this put me in tension with many of my colleagues in CS, who have been unequivocally optimistic about the ability of CS to change the world. With many of my newest doctoral students embracing critical perspectives of CS, and honestly, my recent acceptance of my own personal position as as a trans woman oppressed by a myriad of mundane software design choices, it was about time I amplified these ideas in our work.
As with any new direction, our work has been “productively unfocused.” I did a few papers on informal mentorship with my iSchool colleagues, exploring injustices in who has access to mentorship, and what impact that mentorship has, finding that mentorship is surprisingly central, even to students underrepresented in computing, and marginalized by structural inequalities in access (Ko et al., 2017; Ko et al., 2018). My Ph.D. student Kyle Thayer and I explored structural barriers to engaging in coding bootcamps, finding that they replicate many of the same injustices in higher education, while creating new ones (Thayer & Ko, 2017). My Ph.D. student Alannah Oleson and I began investigating the teaching and learning of inclusive design and how design as a practice is entangled with CS education in messy ways (Oleson et al., 2020). My former student Yim Register and I began investigating machine learning literacy, defining it not as the ability to create with machine learning, but to advocate against the injustice amplified by machine learning (yet to be published). I also have students investigating culturally responsive computing, the implicit norms (and capitalist forces) that shape students’ career choices. And I’ve been (trying) to fundraise on a vision of preparing CS teachers that educate youth about both the powers and perils of computing.
These few years of what I would personally describe as scholarly meandering is just beginning to coalesce into a more coherent trajectory. I gave a keynote this last November expressing one vision for what a more critical CS education would look like. I’m preparing an even more focused version of the keynote for a future ACM Inroads article. And I hope, with some luck, that NSF and my peers grant me the resources to continue down this path, envisioning a more balanced vision for teaching computing as something both powerful and perilous.
Reflection
It always surprises me how little I can predict where I’ll go next as a scholar. Ideas mature and enter the world; we read, reflect, and discuss, and ideas change. A lab isn’t just a bunch of people writing papers, it’s a microcosm of the larger scholarly discourse, taking in ideas, reshaping them, and resharing them. In all three of the threads above are decades of research from entire research communities, plus a bit of new insight from my students and myself. All I can hope is that those new insights have the same effect on other scholars that their ideas have had on us!
