How should one year of academic work be judged?
Most universities have some way of evaluating their faculty, even after tenure. We’re often asked to self-evaluate, evaluate each other, or be evaluated by some established set of rubrics. These merit review policies vary in their utility; some self-reflection processes can be quite revealing. Others can destroy morale.
I’m fortunate to be in a school that has a process with some helpful elements. It’s not perfect, but we have a culture that always drives improvement. One of our practices is to write “narratives” that synthesize 12 months of effort in research, teaching, and service. We then spend some time reading the narratives of the faculty below us in rank (with full professors’ reading other fulls’ narratives). We’re about to do this process in a couple weeks.
I like the narrative part of our process. I get to learn about what my colleagues have been up to, and calibrate against the different norms across the various fields in our interdisciplinary school.
Here are the narratives I wrote for research, teaching, and service for 2017. As you read them, notice how the boundaries between research, teaching, and service are often blurry.
Research
Because our process is tied to merit raises, my narratives about my research are all about convincing my colleagues that I’ve had a super-productive-unparalleled year of scholarly progress. That makes my narratives come off as unvarnished gloating, which always feels a bit icky to me. At the same time, a punchy summary of highlights may be the only exposure my colleagues get to my work. Here’s what I wrote this year:
In 2017, I fully recovered from being on leave at AnswerDash for 2 years and pivoting to the new research area of computing education. My lab reached my desired capacity of six doctoral students, I reattained my regular pace of publishing, and after the expiration of all of my sponsored research grants, I raised all of the funding necessary to keep my lab at full productivity for 3 years. I also began a substantial and intentional effort to disseminate my research through weekly blogging and systematic efforts at bringing my discoveries to relevant audiences.
Here is 2017 by the numbers:
- As lead or co-PI, I raised $1.75 million in NSF funding. I also helped co-PI an NSF Expeditions in Computing proposal, the largest propose that NSF’s CISE directorate gives. Our proposal was invited to the final round of full proposals out of more than 100 preliminary proposals.
- My lab published 9 full peer-reviewed papers at the top HCI, Software Engineering, and Computing Education conferences, including one CHI best paper nomination.
- My publications were cited a record 600 times across 5 distinct research areas of computing education, HCI, software engineering, programming languages, and learning science.
- Since I started blogging in May 2017, my weekly posts have been read viewed more than 100,000 times. This does not include a guest post by my student Kyle Thayer on our coding bootcamp work, which was read by over 100,000 additional people via Reddit.
- I was invited to give talks at HCI seminars at 3 peer institutions (Stanford, Northwestern, and Michigan), and keynotes at 2 top conferences (ACM SPLASH, ACM ICSE).
- I created the Sound CS Ed community, spanning 80 researchers, product designers, policy makers, and teachers passionate about effective, equitable, and scalable computing education in Washington state.
- I was invited to serve in 2 leadership positions: the international ACM Education Council and the Washington State Governor’s K-12 CS Education Advisory Board.
The summary is useful to me emotionally, because it validates the stress I’ve felt this year (as in, “holy crap, no wonder I’ve felt busy!”). I’m afraid that it comes off as wanting to sound busy though, even though that’s not my goal. I am busy, because I’m just curious and motivated. I just like doing research!
Teaching
Whereas the research narrative sounds explicitly boastful, my teaching narrative reads a lot more like humble-bragging, because teaching is inherently an act of service to others. It’s basically “Look at all this stuff I did for other people.” And yet, it’s also intended as useful information for my colleagues; it helps them see how I spend my time, and more importantly what I value. It might also shape what they value. Here’s what I wrote:
Last year, I brought much of the new expertise I acquired on learning to my classrooms, and saw many improvements in learning outcomes and student experience. In 2017, I tried to share my new expertise in three ways: creating robust, easily adoptable course materials, writing accessible books for my courses, and mentoring numerous junior colleagues and teachers on teaching.
For course materials, my efforts involved creating materials for INFO 360, INFO 461, and HCID 520 that anyone in the world could view and adopt in their classrooms. The key part of this was going beyond the content students view, also adding materials for instructors to help them understand the rationale for the course design and how to adapt the ideas to their own classrooms. The result is that many of the INFO 360 instructors are using my materials and dozens of other HCI educators in the world are reusing them in their own classes. The same has been true for my software engineering class, INFO 461, which is now being used at over a dozen institutions.
My second teaching effort has been writing online books for my three courses that are web accessible, engaging to read, and up-to-date with the latest research:
I write these books as alternative to lectures in class. They also have the benefit of being free, engaging surveys of research that can be accessed by anyone in the world. I view these as living documents, synthesizing the latest research from conferences and journals every year, keeping the content fresh. In 2017, they were read by more than 1,000 students, teachers, designers, and engineers across academia and industry.
My third teaching effort has been mentoring. I’ve worked weekly with numerous people in and out of the iSchool. It also includes doctoral students who have been teaching in the iSchool. In my role as Informatics Chair, I’ve also mentored 25 guest faculty, helping them onboard and prepare to teach in our Informatics program.
In addition to the narrative above, my teaching evaluation also included my student course evaluations (which were 4.3, 4.8, and 5 on a scale of 5), plus an entirely new holistic teaching self-evaluation that we have started including as an optional alternative to the inescapably biased student evaluations. What’s cool about these is that they are constructive, providing normative guidance on expectations in teaching, and ways to improve.
Service
Many faculty despise service. It doesn’t serve their intellectual passions, and rarely relates to their area of expertise. My service in 2017 was dominated by administration, which is even further removed from my core interests. And so my narrative was much more about describing some of the mundane, but incredibly important transactions, resource allocations, and process improvements I’ve. (There’s also an implicit hinting of “Help, I’m drowning in service!”). Here’s what I wrote:
In 2017, I was chair of both MHCI+D (until September 2017) and Informatics (beginning in April) with only one course release. It was only barely possible to maintain both of these roles simultaneously, so my efforts were quite focused on critical needs.
For MHCI+D, my goals were to hire a new director for the program, find new studio space, renew our MOA for the program between CSE, HCDE, the iSchool, and Art+Art History+Design, and find a new chair to replace me. I hired Michael Smith from Frog as our new Director, helped find and sign a lease for the first floor of the UW Alumni House for our design studio, got signatures from all stakeholders from our new MOA after several rounds of discussion and feedback, and recruited Jeff Heer to chair MHCI+D as my replacement.
For Informatics, I’ve focused on taking a role that Scott Barker had structured as a full-time job and reducing it to a 20% time job. This has involved streamlining processes, focusing admissions, delegating more heavily to staff in student services and academics. I’ve also made some incremental changes to curriculum, to capstone, and developed some new partnerships around a shared data science course on campus and a shared new option with BIME.
Apparently I forgot to mention the 30 or so research papers I reviewed on program committees and journal editorial boards. That was too many.
My commentary above on my narratives above might come off as a bit cynical. I do feel cynical to some extent: I think the purest idea of academic involves unbridled, free pursuit of ideas, and evaluating the “productivity” of this endeavor feels like a straightjacket in the face of intellectual freedom.
And yet, I really value the self reflection. It helps me calibrate my time, it helps me feel some pride in my accomplishments, it’s one of the few ways my colleagues see my work, and it’s a useful social process for constructing and revising the norms in my academic unit.
Are you faculty? How does your unit evaluate you? Do you find it constructive? If you’re not faculty, how does your employer evaluate you? We revise our process every year, so I’d love to hear evaluation ideas that you find constructive.
