My view from Monte San Salvatore, south of Lugano, Switzerland (a 2.5 hour hike).

ICER 2022 trip report: Together again, as bits and atoms

It’s hard to believe that just one year ago I was co-organizing ICER 2021 (and just two years ago, co-organizing ICER 2020). In many ways, those two years feel like an entirely different era of my life: I spent 98% of my time at home in the exact same room in my house, which served triple duty as an office (for work), a bedroom (for the hot summer nights), and an exercise studio (to keep moving), and for several months, a recovery room, as I healed from surgery and regained my strength. It seems impossible looking back that I helped plan two events, three times over (as collocated, hybrid, and eventually virtual events), in that same room.

And so the last time we were in person for ICER, in Toronto, Canada, seems like a lifetime ago. August 2019 was a pivotal time in my life: I was very much out as trans to myself and my wife, and deep in planning how I’d come out to my communities. And yet, I wasn’t out, and so spending a week in Toronto pretending just one more time to be a man, wasn’t the most comfortable of experiences. It was surreal, in a way: I remember being alone in my hotel room and out on walks in the city feeling such a strong sense of peace, relief, and resolve that I’d chosen to be me that attending the conference almost felt like going back into time, into a cage, where I had to hide. The only thing that made it okay was knowing that just a few weeks later, I’d tell all of the folks I’d been lying to the truth.

And so returning in person to ACM ICER 2022 is something I’ve anticipated for three years, both to escape my room, but also to return to my community as myself. Of course, everyone knows me as me by now, at least virtually, and so it was bound to be anticlimactic. But having spent the last year cautiously traveling to other conferences, I also knew how much seeing people again in person would bring some much needed closure to a long transition.

By now, some of you are probably thinking, “Ummm, wasn’t this post supposed to be about computing education?” Yes, we’ll get to that :) But isn’t all of the above about computing education? Why did it take me 25 years of being in CS to feel safe coming out? Might it have something to do with the fact that I only ever met one other trans person in CS, and they were at the margins? Or that conversations about this marginalization didn’t really even start until the 2000’s? Or that those conversations were broadly framed around binary conceptions of gender, surrounded by vaguely essentialist notions of interest and ability? Computing education is not just about logic, or formalisms, or programming; it as about people and learning, and therefore every aspect of the social contexts in which people learn. And my identity — and visibility—is as much a part of that fabric as any scientific insight about cognition, transfer, or functions.

And so in addition to eagerly anticipating reconnecting in Lugano, Switzerland, I also was excited to join our small but mighty community to remind everyone—by my presence and words—that identity, equity, inclusion, diversity, and justice are as much essential topics in computing education as are our more technical concerns.

Prior to coming to Lugano, I spent Friday and Saturday in Zürich on a bit of holiday. I hadn’t been since visiting ETH Zurich about 3 years prior, also before I’d come out. Returning brought all of the same feelings I’d had prior: a deep respect for its order, its public infrastructure, and its immensely walkable streets, but also a foreboding threat of conservatism. After all, my marriage to my wife only became recognized one month before I arrived. I read horror stories of trans and queer folks being beat up on the street at night, and so proudly strolled the streets with my pride watch band with caution.

Fortunately, I was not assaulted (!), and safely caught a train to Lugano Sunday morning after enjoying a very tasty bratwurst.

Matthias shares the local history.

Sunday

After settling into my Airbnb (add oddly placed tiny house behind the Impact Hub incubator villa), I went to the meeting location and visited with the organizers, who were doing final testing for the hybrid configurations. We then went on a walking tour of Lugano on our way to the evening reception at LAC. I had a wonderful few hours of conversation with many I haven’t seen in two years, as well as many students who are new to the community. On our way out, we all got stuck in an intense thunderstorm downpour. I waited it out in a covered bistro outside the gallery with a few attendees and we had some nice drinks and toast snacks.

Matthias describes our pans for hybrid.

Monday

In the morning, I found a light breakfast at a small local restaurant, then head to the conference opening on the The Università della Svizzera italiana campus. The chairs talked about the hybrid adventure we were all about to embark on, which included ~110 in-person attendees and 45 virtual attendees, placing it at about the same size as the last ICER pre-pandemic. The conference also accepted 25 research papers for presentation, along with the usual lightning talks, posters, doctoral consortium, works-in-progress workshop, and a keynote. The organizers did a great job envisioning a seamless hybrid experience, including fun nudges to get in-person attendees to engage with virtual attendees online. I was excited to see how it would play out!

Julie smith gives her lightning talk.

We started the morning with lightning talks, some of which were also posters, and a mix of in-person and hybrid. As in previous years, the lightning talks were a rich cross section of the field’s interests, spanning the history of computing as a scaffold, IDE support for misconception recognition, intersections between computing education and data science education, associations between spatial skills and learning outcomes, machine learning education through interactive scaffolded activities, action research on ethical work in industry, and the impact of computing on mathematical creativity. Afterwards, we all split off to physical posters in a poster area, virtual posters with cookies with online presenters, and of course, hallway conversations completely unrelated to posters and lightning talks over the coffee break. The organizers offered a generous hour to reconnect and talk before launching into paper sessions.

Miranda explains an acronym.

After a wonderful hybrid break chatting with in-person and remote attendees, we had the first paper session on programming assessments. Here’s a quick synopsis of the three:

• Miranda Parker (San Diego State University) presented “A Pair of ACES: An Analysis of Isomorphic Questions on an Elementary Computing Assessment”. She talked about the careful redesign of an assessment for primary students about loops and sequences, creating additional isomorphic questions. Interestingly, and importantly, changing the assessment’s illustrations did not result in differences in scores, but changing agent movement direction did create different difficulties, possibly because of interactions between spatial reasoning and language literacy, but I’d imagine also cultural knowledge.
• Xinying Hou (University of Michigan) presented “Using Adaptive Parsons Problems to Scaffold Write-Code Problems”. Across two studies, she explored opportunities to use inter- and intra-problem scaffolding to support the development of program writing skills. She found that students found scaffolding to be helpful for getting started, for finding hints to translate their ideas into code, to help localize faults. A controlled experiment demonstrated that the scaffolding reduced practice completion time, but differences couldn’t be observed due to a ceiling effect in the pre-post assessment.
• Juho Leinonen (Aalto University) presented “Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models”. He discussed a method of creating items automatically, using a combination of templates, Codex, and single shot learning to generate program explanations. The team found through an expert analysis of generated items that most items (~70%) were sensible, novel, and well aligned with the provided themes, domains, and concepts, suggesting their utility as an aid in a more elaborate item design process, but certainly not a replacement for expert item design.

Jayce Warner presents an empirical analysis of Texas public school data.

After a lively lunch in the cafeteria, we had another paper session on diversity, equity, inclusion, and belonging.

• Sarah Narvaiz (University of Tennessee) and Lexi Lishinski (University of Tennessee) presented Self-efficacy, Interest, and Belongingness — URM Students’ Momentary Experiences in CS1. Using an experience sampling method, they specifically investigated how students from groups underrepresented in CS have shifts in their momentary sense of belonging over time. They found a variety of differences and interactions, but I found the interactions between self-efficacy and final grades particularly interesting, suggesting that students experience failure differently depending on how marginalized they might feel. I wondered how much this was mediated by stereotype threat.
• Jayce Warner (UT Austin) presented Gender, Race, and Economic Status along the Computing Education Pipeline: Examining Disparities in Course Enrollment and Wage Earnings. They quantitatively examined multiple dimensions of disparity as predictors of course enrollment majoring, and wage earnings, while account for multi-marginalization. With their large sample of students from the Texas public data system, they found that the biggest reduced likelihood factors were girls of all racial identities, but especially so for Black and Latina girls.
• Spencer Yoder (North Carolina State University) presented Gender, Self-Assessment, and Persistence in Computing: How gender differences in self-assessed ability reduce women’s persistence in computer science. Through survey and interview methods, they examined an introductory programming course. They found that women at the institutions studied self-assess lower despite having comparable ability but come with higher rates of persistence (likely survivorship bias). Many of the reasons women reported for these trends were because of experiences of disrespectful treatment with presumed incompetence, but also held higher standards of learning than than their male peers.

After talks, we had another lightning talk, poster, and coffee break session. The talks covered self-directed practice environments, teaching variable naming, assessing data science knowledge, gratitude journaling, student professional development, teaching algorithmic bias, culturally sustaining data structures teaching, and conceptions of “good programmers”.

Maria opening her talk on argumentation.

The last session engaged programming problem solving:

• Elijah Rivera (Brown University) presented Plan Composition Using Higher-Order Functions. He talked about the importance of representation in planning using the domain of floor plans, then suggested that higher order functions are one helpful set of primitives for planning, especially in data science. To assess the viability of this, they explored how much students understand the language of higher order functions and how much they could plan and implement with them. They found that students generally understood the logic of higher order functions, that they could form correct non-executable plans.
• Jamie Benario (Google) presented Using Electrodermal Activity Measurements to Understand Student Emotions While Programming. They examined students’ self-reported emotions and the events they attributed to them. They found 21 distinct events that triggered emotions, including things like halted progress, finding a resource, observing a failure, fixing a defect, or simply not knowing something. There were notable shifts from planning to implementation phases, associated with a major reduction in stress response.
• Maria Kallia (University of Glasgow) presented When Rhetorical Logic Meets Programming: Collective Argumentative Reasoning in Problem-Solving in Programming. She gave background on the philosophy of persuasion (Aristotle’s pathos, ethos, logos), on Toulmin’s deconstruction of human reasoning and formal logic, and on Walton’s argumentative reasoning patterns. Maria and her collaborators applied this to examining how programmers of varying expertise engaged in argumentation during group programming: experts used collective and altruistic arguments, graduate students collective but egoistic arguments, and novices with collective “monologues.”

A group selifie at the Cathedral of Saint Lawrence.

After the session, the day was officially over, and we all informally gathered for dinners and evening adventures. I had a lovely dinner with Eliane Wiese and Jason Wiese at by the water at Antica Osteria del Porto, by the mouth of the river that feeds Lake Lugano, then had a warm and quiet walk back to my tiny house.

Dustin talks about undergraduate research misalignments.

Tuesday

The next morning, I had a tasty croissant and espresso at Caffe Ceresiana, then head to the conference for a morning session on undergraduate research pathways. There were just two papers in this session:

• Katherine Izhikevich (UC San Diego) presented Exploring Group Dynamics in a Group-Structured Computing Undergraduate Research Experience. She explored a hypothesis that grouping students might help improve peer support and scale. The particular grouping strategy involved careful formation of peer mentors and grouping mechanisms. Not surprisingly, creating a positive social context for sharing was broadly helpful, especially to students marginalized in some way by their identity. There were continued struggles, however students who continued to fear sharing their knowledge gaps.
• Rhea Sharma and Dustin Palea (UC Santa Cruz) presented “It’s usually not worth the effort unless you get really lucky”: Barriers to Undergraduate Research Experiences from the Perspective of Computing Faculty. They interviewed 12 faculty about their research mentoring experiences, surfacing misalignments that created barriers to students participating in research. The faculty generally reported that their research goals were in tension with student goals, knowledge, availability, and expectations, though it wasn’t quite clear what incentives these faculty were under or how they organized their research activities.

My student Alannah gives their lightning talk.

After the short session, we shifted to another round of lightning talks, posters, and coffee break, this time centered on the doctoral consortium students. The students topics included developing learning theories with programming process data, intelligent programming tutoring systems, individualized instruction, program comprehension, quantum computing education, productive failure pedagogies, HCI education, teamwork tools for capstones, and predicting learning outcomes from student perceptions.

Kai presents on teamwork struggles.

The session after the break was on groups, teams, and social learning.

• Lauren Margulieux (Georgia State University) presented Getting By With Help from My Friends: Group Study in Introductory Programming Understood as Socially Shared Regulation. The work was framed around socially shared regulated learning, and examined how groups facilitated regulation. They examined ~1,000 students’ self-expressed learning strategies in the context of opt-in study groups in Auckland, NZ, finding that some of the more prevalent co-regulation phenomena was social help seeking.
• Kai Presler-Marshall (Bowdoin College) presented What Makes Team[s] Work? A Study of Team Characteristics in Software Engineering Projects. They examined the many dimensions of teamwork and project management from the students’ perspectives and strategies that students used to overcome teamwork challenges. In general, they found many of the same challenges reported in poorly managed and organized software organizations such as weak leadership, misaligned incentives, and poor communication.

John wraps up his talk on notional machines about call stacks.

After wonderful lunch of roast turkey, I returned for another short session on notional machines:

• Gayithri Jayathirtha (University of Oregon) presented “How does the computer carry out DigitalRead()?” Notional Machines Mediated Learner Conceptual Agency within an Introductory High School Electronic Textiles Unit. She examined how teachers designed notional machines to help students reason about program execution, discovering many interesting strategies, such as notional machines that would reveal naïve conceptions to help deepen them.
• John Clements (California Polytechnic State University) presented Towards a Notional Machine for Runtime Stacks and Scope: When Stacks Don’t Stack Up. John specifically examined how to represent call stacks; they found that students understand that call stacks contain bindings of names to values, but there were divergent views of call stacks as regions of memory versus data structures, many students did not pop things off the stack when simulating execution, and many students ignored execution context.

Manu starts his talk.

After a coffee break, we had a keynote from Manu Kapur (ETH Zurich), who spoke about productive failure. Manu is a learning scientist, but also has a background in engineering (and football!), and has led many programs and centers related to learning sciences and currently runs a center engaged with a full spectrum of learning sciences research, both basic and applied. His keynote focused on the phenomenon of initial learning — the process of coming to understand something new, especially abstract things like mathematics or computing. He motivated this focus with a paradox: pedagogy that is perceived to be great can produce poor learning outcomes. He summarized research on why this happens as follows: experts see very different things than novices, and typically the deep structures of phenomena. He gave another example comparing two groups, one that receives no guidance on how to play with a toy and one that receives explicit guidance: the former is more creative (e.g. functional fixedness).

With this setup, he turned described productive failure as the idea deliberately designing for failure in order to intentionally create opportunities for feedback and learning. For example, this might involve creating a stats problem where the common student strategy that does work for some problems doesn’t work, in order to surface brittle conceptions. In these contexts, if tasks are designed carefully, students can quickly converge towards big concepts in statistics, such as standard deviation. In one experiment (and 166 replications), he showed that direct instruction followed by problem solving was more effective than the reverse at producing conceptual understanding and near transfer (with a moderate effect size). He discussed some of the key elements for how and why this works, such as activation, showing that observing failure vicariously is less effective than youth experiencing failures themselves. Problem finding was also better at promoting transfer than productive function, but less effective at promoting conceptual understanding. He did find that domain general skills (designing experiments, reasoning about causality) did not benefit from productive failure.

Xiaohua describes her research questions.

The last session of Tuesday was on teachers.

• Xiaohua Jia (Leiden University) presented Teaching Quality in Programming Education: the Effect of Teachers’ Background Characteristics and Self-efficacy. She examined 164 teachers use of teaching methods in programming courses in Chinese schools and how these practices were mediated by self-efficacy and knowledge. They found that direct instruction and classroom management were the most common teaching skills enacted, and that different degrees of quality in these were related to teacher‘s self-efficacy.
• I presented my Ph.D. student Jayne Everson’s paper “I would be afraid to be a bad CS teacher”: Factors Influencing Participation in Pre-Service Secondary Computing Teacher Education. We explored the factors that shape teacher candidate’s willingness to pursue a CS pre-service pathway and found that many surprising factors deterred CS teaching, including fear of extra teaching preparation, a fear of lack of respect, and a fear of perpetuating poor quality CS instruction they had experienced in their own education. But we also found motivating factors, such as correcting this historical injustices in their CS learning, empowering youth with what they viewed as a critical literacy, and connecting CS with other disciplines.

My lab does a group selfie.

After the session, the community walked to the funicular (cable car) for a sweaty ride up the mountain to the beautiful vista of Monte Brè. Company was good, dinner was tasty, and the views of the lake and surrounding hills and valleys were outstanding. We didn’t do most of the ICER traditions — an organizer performing or ICER trivia—but from the constant chatter at the distanced tables on the patio, I think everyone engaged the community’s consistent practice of conversation and witty banter.

Diana shows an example of a student plan.

Wednesday

The last day of the conference was a focused series of three paper sessions, followed by an awards session and conference reflection. The first session of the morning was about primary and secondary computing education in schools.

• Diana Franklin (University of Chicago) presented Investigating the Use of Planning Sheets in Young Learners’ Open-Ended Scratch Projects. She talked about the tensions of 11–14 year olds’ ambitions in open-ended project-based learning, and the merits of planning documents to try to bound ambition to learning objectives. In a comparison of classes and school years with 103 students, Diana’s team found that when youth are expected to make plans, they do make them and do implement most of them, but that many students don’t implement interactivity, but that requiring plans was associated with meeting more project requirements. (The study didn’t reveal reasons for these trends).
• Jane Waite (Raspberry Pi) virtually presented What do We Know about Computing Education for K-12 in Non-formal Settings? A Systematic Literature Review of Recent Research. They used the UNESCO definition of “non-formal”, as institutionalized, intentional, and planned education by a provider that supplements or is an alternative to formal education (and so they were not talking about informal learning). Their analysis of 6 years of studies found that most studies focused on interest, perception, engagement, and self-efficacy effects with the goal of broadening participation and a strong (likely publication) bias toward positive effects.
• David Gonzalez-Maldonado (University of Chicago) unexpectedly virtually presented Comparison of CS Middle-School Instruction during Pre-Pandemic, Early-Pandemic and Mid-Pandemic School Years. David’s team examined changes in the delivery of the Scratch Encore curriculum in pre, early, and mid-pandemic school years by examining data from Scratch projects and partner school districts. They found that after lockdown, Scratch activity essentially disappeared, but later, it recovered to baseline levels mid-pandemic. Pre- and mid-pandemic auto-grading performance also returned to baseline.

Alannah and Benji presenting online after a positive COVID test forced many virtual.

After a lovely break with some virtual attendees, I returned to the next session on reviews of computing education research.

• Monica McGill (CSEdResearch.org, Knox College) Joey Reyes (Knox College) presented Surfacing Inequities and Their Broader Implications in the CS Education Research Community. They explored equity barriers that different researchers face conducting computing education research through a systematic literature review and survey. Most of the factors they found were capacity issues, including lack of funding, collaborators, publication venues, participants, tools, instruments, job prospects, time, and a sense of belonging, recognition of and respect for the discipline.
• Neil Brown (King’s College London) presented Launching Registered Report Replications in Computer Science Education Research. He described a shared concern that our research would not replicate and described questionable research practices such as practices that lead to publication bias, manipulating data to “p-hack”, discarding hypotheses in favor of post-hoc significant results. He then described and advocated for pre-registration as a way of preventing questionable practices. I publicly committed to implementing pre-registration for ACM TOCE.
• Alannah Oleson and Benji Xie (University of Washington) presented our second ICER paper, A Decade of Demographics in Computing Education Research: A Critical Review of Trends in Collection, Reporting, and Use. We examined how researchers are categorizing people demographically and how that reinforces power, hierarchy, and hegemonic norms. The paper examined how data was gathered, reported, and used, finding that papers generally are quite exclusionary and ambiguous in their reporting and use of terminology, especially with aggregate terms like “at risk”, “underrepresented”, and “non-STEM”.

Noelle introduces her title slide and topic.

After a quick lunch, I returned to the last paper session on responsible computing.

• Noelle Brown (University of Utah) presented The Shortest Path to Ethics in AI: An Integrated Assignment Where Human Concerns Guide Technical Decisions. She described an approach to tying ethical considerations to the specific technical details of algorithms, particularly in graph search algorithms and examined how assignments might simultaneously assess technical and sociotechnical concepts. Their (very clever) assignment design challenged shortest path algorithms with a different cost function around someone traveling a path with a chronic condition. The core idea was situating technical decisions in a specific social context, and assessing the extent to which solutions considered both technical and sociotechnical dimensions of the algorithm and social context.
• Amreeta Chatterjee (Oregon State University) presented Inclusivity Bugs in Online Courseware: A Field Study. They examined CS courseware for inclusion flaws using a novel automated tool based on the GenderMag method. They demonstrated that the tool was able to find many of the inclusion problems that an expert could, and that faculty using the tool independently were able to use the tool to find some of these problems, but that faculty sometimes disagreed with the tool’s analyses.

Jan thanks the student volunteers for running a great hybrid event.

After these last two talks, we had a closing session. It began with the honorable mention, best paper award, and lasting impact award announcement. The awarded papers were:

• Best paper. Sami Sarsa et al.’s Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models, for its bold exploration into the opportunities and limits of language models for supporting assessment design.
• Honorable mention. Maria Kallia et al’s. When Rhetorical Logic Meets Programming: Collective Argumentative Reasoning in Problem-Solving in Programming for its critical analysis of student argumentation about problem solving in social settings.
• Lasting impact. Andy Begel and Beth Simon’s Novice Software Developers, All Over Again (ICER 2008), for its rich insights into “soft” skills in software engineering and their critical absence in most computing curricula.

Reflection

As an in-person attendee, this felt like the pre-pandemic ICERs I’m used to. But it also felt new: we had dozens of virtual attendees and genuine interactions with them during breaks and on Discord. We talked about new topics, ranging from critical perspectives on computing to the disruptive changes that may come about from advances in machine learning. And for me personally it felt new, because I finally felt safe enough to share myself as I am, rather than pretending to be someone else. I felt like this freed to me connect with people far more than I have before, which felt wonderful, necessary, and quite belated.

But it wasn’t perfect. Yes, there were the usual hybrid hiccups, but the real problems stemmed from ignorance, assumptions, and stereotypes: my students reported being frequently misgendered, one was cornered and forced into uncomfortable conversation by a senior faculty member, and the community is still full of harmful stereotypes about students, culture. For anyone marginalized in CS or in the world more broadly, I wouldn’t say that the broader computing education community, or the subset that comes to ICER, is a completely safe space yet. In future years, I’ll likely need to talk more purposefully with my students about the likelihood of microaggressions and harmful ideas they’re likely to encounter as the broader community slowly learns to see and accept the diversity of people in the our community in and schools.

All that said, I’m optimistic. Our community is talking about diversity regularly and many are doing the work on themselves to move past ignorance. It’s going to be a while before ACM ICER, SIGCSE, and ITiCSE and have the same level of diversity literacy has those that tend to attend IEEE RESPECT and CSTA, but it will happen. I won’t let it be otherwise.