On the academic quantified self
This spring I’m preparing my materials for promotion to full professor. Our process at the University of Washington, like most institutions, involves preparing an updated curriculum vita, but also a set of research, teaching, and service statements that summarize the entirety of my academic career. These documents are the basis for both internal and external evaluation my scholarly achievements to date. And if the senior faculty in my academic communities find my achievements sufficiently meritorious, I earn the title of Professor, an increase in salary, and a host of new rights and responsibilities. It’s one of the highest distinctions available to academics, and something that takes an entire career to achieve.
In preparing these documents, however, I’ve begun to notice a duality in how we expect academics to articulate their achievements. In one way, the documents are a cataloging of my achievements: my job is to list every publication, every award, every course, every act of service, and every notable impact in my career. This bean counting is one of the driving incentives in modern academic work, and a dominant frame of evaluation: how many awards did I get, how much money did I raise, how did students rate my teaching? There is a semblance of objectivity in these quantitative forms that belie mountains of nuance, but they’re quick to consume and judge.
Whereas these quantitative forms summarize my work, my promotion statements are simultaneously a narrative of my scholarship. They describe a long-term interest, my efforts to pursue it, the rich intellectual discoveries that have emerged from this pursuit, and the impact in academia in the world that these discoveries have had. This narrative, which is inherently qualitative, is fundamentally about the ideas that have shaped my thinking, the ideas I have created, the ideas that my ideas have informed, and the trajectory of those ideas into society through academic discourse, teaching, and broader dissemination.
In writing these promotion statements, it’s become crystal clear that one of the quantitative and qualitative forms of my academic accomplishments are not equally important. My CV is long; the number of accomplishments it lists is large. And yet that number fundamentally means nothing. What really matters is that the ideas I develop are powerful, true, and impactful in the world, and that people use them. The cataloging and quantification of this impact is really just a fuzzy approximation of this impact; the narrative is a much richer and more precise articulation of this impact, though harder to comprehend because of its nuances.
And yet, while academia’s long-term incentives such as promotion are aligned with this narrative of impact, academia’s short-term incentives are the exact opposite. We worry about the next paper, the next grant, the next unit of progress because this is what we think our institutions are tracking. Only with a relentless focus on where these incremental steps take us does any of it add up big, meaningful ideas. Our institutions fail to measure these long term goals and progress toward them, instead focusing on more easily measurable indicators of merit. This has created an academic world in which academics have been reduced into academic quantified selves, empirically self-tracking our short-term units of progress rather than our long-term collective participation in the development of powerful ideas. In this narrower conception, we forget that we are scholars, holding the highest integrity of intellectual freedom and honesty in the lifelong pursuit of knowledge.
How can senior faculty like myself model scholarly selves rather than quantified selves? One small way is rethinking how we talk about our careers. For example, academics often write short biographies for presentations. Here’s the one currently on my website:
Amy J. Ko is an Associate Professor at the University of Washington Information School and an Adjunct Associate Professor in Computer Science and Engineering. She directs the Code & Cognition Lab, where she invents and evaluates interactions between people and code, spanning the areas of human-computer interaction, computing education, and software engineering. She is the author of over 80 peer-reviewed publications, 11 receiving best paper awards and 3 receiving most influential paper awards. In 2013, she co-founded AnswerDash, a SaaS company offering instant answers on websites using a selection-based search technology invented in his lab. In 2010, she was awarded an NSF CAREER award for research on evidence-based bug triage. She received her Ph.D. at the Human-Computer Interaction Institute at Carnegie Mellon University in 2008. She received degrees in Computer Science and Psychology with Honors from Oregon State University in 2002.
This is very much a quantitative framing of my academic accomplishments. It lists paper counts, award counts, titles, and distinctions. To those who aren’t thinking very deeply about my career, it might sound impressive, but it doesn’t really say much about my ideas or their impact.
Here’s a more scholarly version, eliminating all titles, distinctions, and metrics, and instead focusing only on my ideas:
Amy J. Ko has studied the cognitive and social aspects of programming since 1993, since she was a curious adolescent fascinated by the expressive powers of computing. Her earliest discoveries concerned how people can partner with software development tools to find and fix software defects; some of her most impactful ideas include a new genre of tools that allow programmers to directly express the questions that that they want to ask about their software. Her later work expanded in scope, considering the interaction between programmers and the potentially billions of people who use what they create; the ideas that emerged from this work conceptions of the encoding and aggregation of people’s experiences with software into actionable data. Her most recent investigations have conceptualized the skills involved in programming, theorizing about the interplay between rigorous knowledge of programming language semantics, strategies for addressing the range of problems that arise in programming, and self-regulation skills for managing the selection, execution, and abandonment of strategies; these are impacting how programming is learned and taught. Throughout all of this work, she has drawn upon ideas from software engineering, human-computer interaction, programming languages, psychology, organizational science, learning science, and education. Her ideas have shaped these and other fields.
Which one of these biographies means more? Which one is more informative? Which one is more consistent with the purpose of academia as serving the relentless pursuit of knowledge? I would argue that the scholarly narrative wins in all three cases. The quantified bio is comparatively meaningless, boring, and ultimately irrelevant to the pursuit of knowledge. All it does is serve to signal my merit, which is irrelevant to progress.
Of course, the scholarly narrative above changes quite slowly, which makes it somewhat less useful for the annual merit reviews my university requires. The scholarly bio above would hardly change at all in a year, whereas the quantitative bio would at least have some bigger numbers and perhaps a new award or title. In contrast, scholarly progress isn’t always visible at the granularity of a year, and so we don’t ask for it. And yet, by not asking for it, we set an incorrect expectation that all academia cares about is the things that we can measure on an annual basis. This creates a vicious cycle in which junior faculty optimize metrics, when they should be optimizing their scholarly bio.
While I can’t easily solve this problem unilaterally for all of academia, I can send a strong signal to the junior scholars in my academic communities: long-term, academia doesn’t care about bean counting. Tenure and promotion in all forms is fundamentally about your ideas and their impact. That is why we ask junior scholars to write narratives about their ideas, and for senior scholars to write letters evaluating their ideas. These are the norms we are trying to build into our processes at the University of Washington Information School.
And to my senior colleagues, if the way you evaluate junior scholars isn’t consistent with the above, change how you evaluate. Read the CRA memo on evaluating for promotion and tenure and memo on incentivizing quality and impact, which both clearly delineate the problems with quantifying our academic selves and alternative frames for evaluating scholarly work. Many of you will evaluate a few tenure cases this summer; if you’ve made it this far through this post, perhaps you’re motivated enough to read those memos before you write your letters of evaluation.
As for myself, as I finalize my statements for promotion to full professor, I’ll be keeping this duality in mind, primarily focusing on my scholarship, my ideas, and their impact, and not on the numbers. I hope that whoever reviews my case finds this far more informative (and interesting) than a quantitative account of my scholarship, and easier to evaluate as well.
