HCI peer review in crisis: what is science in HCI?

HCI is in continual crisis mode with regard to peer reviewing. Every year around the time of the CHI conference — or to give it its full, unwieldy and faintly outmoded title The ACM SIGCHI Conference on Human Factors in Computing Systems — there is considerable debate about the reviewing process (see the aside at the bottom of this posting for more detail on the reviewing process itself). No doubt similar conversations take place at all other major HCI and related venues (e.g., CSCW), but it seems that the ones from CHI leak onto the web most readily. Other conferences have comparable review processes (CSCW, DIS, Ubicomp), so the conversation here is relevant beyond CHI.

It seems to me that a lot of these conversations revolve around a number of topics related to the review process itself, such as the relative weighting of reviewer and author power, the accountability of activities of programme (sub)committees, the way reviewers get selected, the rising numbers of submissions to the conferences, and (relatedly) the acceptance rates and submission types. As an author it can be incredibly frustrating if the account of why your paper was rejected is scant on detail (such as reports that your paper was discussed at the committee meeting, but the actual detail of the discussion was left out), and correspondingly as a reviewer it can be annoying to have to review work which does not seem to even hit the lower level of what could be considered worthwhile.

In response to participating in the process of peer review, Jeffery Bardzell recently wrote a very interesting blog posting outlining the various ways in which not only practical changes could be made to this reviewing process, but also presented some ways the HCI community could reconceptualise what process it is actually engaging in during peer review. Bardzell presents this in his “Position on Peer Reviewing in HCI” (in three parts), and formed a lengthy response to Antti Oulasvirta’s original blog posting on “Why your paper was rejected” with specific reference to CHI.

Oulasvirta’s blog here is key in understanding the meaning of the exchange and in assessing some of the underlying concerns that are perhaps helping animate the discussion. Oulasvirta, as a participant in a subcommittee of CHI 2012, summarises some of the key problems that result in low scores being awarded to some kinds of paper submitted to CHI. He focussed particularly on what he terms “flaws in empirical work”, including such things as: errors in research strategy (i.e., applying the wrong research tool for the job); basic statistical flaws (interestingly Grounded Theory is included here related to errors in inter-coder reliability metrics); causation issues (e.g., problems with the ‘implies’ relationship); generalisability and replicability of findings.

Reading the blog post reveals Oulasvirta’s epistemic position on the nature of HCI and its relationship to the natural sciences. While it is not explicitly nailed down anywhere, this position seems to be that HCI (or at least parts of HCI) is a scientific discipline, and as such many submissions to conferences may be shot down on a range of scientific grounds, as outlined above. The notion of HCI as a science is grounded in and guided by popular conceptualisations of ‘what science is / what scientists do’. Specifically these are articulations of what is thought to be the essence of scientific practice, such as: the nature of larger problems is such that they may be broken down into subproblems and reassembled upstream (i.e., scientific reductionism); the mutually reinforcing relationship between theory and experiment, with each elaborating the other — experimentation provides ‘canaries in the mine’ for theory (e.g., see the controversy over string / M theory), and theory directs the ‘what next’ of experimentation; the role of replication and experimentation as separate or separable from context and situatedness. These essences from the natural sciences provide yardsticks with which to measure HCI activity against if it is to be (or remain) a normal scientific activity.

This view of HCI is not new, and often emerges in a range of different guises from HCI research. For instance, Bartneck reports how President of the Association for Computing Machinery (ACM) Stuart Feldman’s CHI 2007 opening speech included the assertion that HCI “is absolutely adherent to the classic scientific method”, which, he said, was “[n]ot a description […] of all the fields in computing”. Ben Shneiderman’s comment on Bardzell’s blog raises this issue again with the comment that “HCI is a scientific discipline”. More recently, the CHI 2012 conference is hosting an ongoing workshop series — RepliCHI — that is concerned explicitly with a cornerstone of the natural sciences, namely, replication. The workshop, expressing very common concerns within the HCI community, states that “it is not uncommon for research to be rejected in CHI for being ‘incremental’”, whereas “[i]n science, it is common for people to try and replicate discoveries, so that the community can confirm new discoveries”. There are other variations on HCI’s scientific status, such as James Landay’s blog posting on “giving up CHI/UIST” (comparable maybe to ‘giving up Christmas’ or ‘giving up on a relationship’?). This is a call for a wider notion of science and where it happens: “I think we have been blinded by the perception that ‘true scientific’ research is only found in controlled experiments and nice statistics”. A now-defunct blog posting by Joshua Kaufman (archive.org link) on a talk by Paul Cairns on ‘HCI as Science’ brings Popper’s conceptualisation of what science is to bear on whether HCI is science (e.g., via Popper’s notion of science as falsifiable). Interestingly one of the commenters argues that HCI is more like engineering: “The Wright brothers constructed a working airplane without knowledge of aerodynamics, fluid mechanics, etc. They just applied tried and true engineering best practices which they learned from working on bicycles (plus a ton of trial and error). In fact, several generations of planes were created before scientists started to grasp any of these principles, and the debate continues today. Yet we fly around in planes. They work, even if we don’t have all the ‘laws’ exactly nailed down.” The unstated assumption being that engineering something that interacts with the physical environment is somehow isomorphic with engineering of things that interact with and around humans (I would argue that they are not).

Bardzell’s blog appears (I think) to be a reaction in some ways to this implicit / explicit epistemological position on the status of HCI as a science. The perceived problems surrounding the peer review process mentioned at the start of this discussion are transposed into or driven by such epistemic issues. While it appears that Bardzell does not deny that there are scientific aspects to the reviewing process, rather than arguing for the scope of what is considered to be normal science to be enlarged (as Landay does), he suggests that we must attend to the social features of reviewing as they play an important (and overlooked) role in the development of scientific results. In sum he appears to be arguing that reviewing is by its nature “critique” more than (or perhaps instead of) being part of a rigorous scientific process that upholds the popular notions of ‘what science is / what scientists do’. (In some ways perhaps it is not much different from film criticism?) This argument obviously borrows strongly from constructionist accounts in Science and Technology Studies (STS) and Sociology of Scientific Knowledge (SSK) (e.g., Latour and Woolgar’s work), where the outputs of the work of the natural sciences is understood in terms of local practical circumstances (e.g., physical lab) typically filtered through the lense of external features such as politics, power structures, funding systems, etc. In my view Bardzell is linking reviewing work with a consideration of these sorts of aspects.

There are a great many interesting aspects of this discussion. However, it turns out that a lot of these concerns are not new, and not particularly unique to HCI. For instance, we can compare issues around peer review in HCI with some of the gripes emergent from SIGCHI’s parent community, i.e., SIGGRAPH. Ashikhmin expresses similar concerns about the review process, fashionable topics, results being presented in a normatively scientific manner in order acquire the cache of validity, reviewer power, acceptance rates, and assertions about computer graphics being a ‘scientific endeavour’. As Ashikhmin states “computer graphics is simply not a hard science and current attempts to present it as such are misguided at best […] the success or failure of any specific technique is, quite literally, in the eye of beholder”.

Another key aspect of the perceived problem is probably the way in which HCI mixes communities (which is a good, but challenging thing). A great deal of this discussion probably culture clash, perhaps between those coming at HCI from a cognitive science and / or psychology background and those from arts and humanities backgrounds (who are increasingly being involved in HCI). While those from such a non-‘science’ background may seek to conceptualise HCI in the mould of the way that the natural sciences are conceptualised in STS, so correspondingly those from the ‘labelled sciences’ (computer science, cognitive science, behavioural science, etc.) are perhaps guilty of mythologising of what the actual work practice involved in the natural sciences is like, or perhaps forgetting that HCI’s related fields like psychology have plenty of their own scientific demons (e.g., problems with WEIRD behavioural and brain science). While I would not necessarily want to align with STS, SSK, etc., particularly as they have a tendency to eschew studies of mundane practice and revert to something like structural functionalism (as noted by Mike Lynch in Scientific Practice and Ordinary Action), there is something to be said for the importance of the social features of scientific practice and its impact on the presentation of science. However, there is probably an equal tendency by those from a non-‘science’ background to presume that such social aspects are somehow hidden from scientific practitioners, and that they are unknowing ‘cultural dopes’ going about their work.

A nice example of the work of a discovering science is provided by Eric Livingston in his book Ethnographies of Reason. Through a simple procedure he unpacks the embodied and commonsense ways in which a phenomenal field may be investigated through various instruments employed to uncover it. In this account, natural phenomena are explored through practical action, and subject to everyday mundane practicalities. This leads to another issue: that of the relationship between the phenomena that are being investigated by HCI and the role of those self-same phenomena within HCI. As pointed out here in relation to Kaufman’s blog posting “HCI phenomena are constantly changing, HCI is constantly moving into new domains, redefining itself and absorbing new types of technology. Basically, there are no static phenomena so there can’t be an HCI paradigm. Furthermore, since there is no HCI paradigm, HCI is not a science.” This is perhaps not deep enough, though. In this situation we can consider the standard ethnomethodological critique. HCI’s phenomena of interest are actually the phenomena that enable us to study the phenomena in HCI in the first place. In other words, the commonsense methods of local, situated reasoning that are brought to bear in the work of HCI research often remain largely unexamined in spite of the fact that really this is the phenomena we are examining when we do a study. While not attending to these issues may be okay for the natural sciences, which may suspend interest in such matters in favour of transcendental principles of the natural world, for the human sciences (which HCI regularly brushes up against, being concerned with ‘interaction’), we ignore these issues at our peril.

Finally there is the legitimacy argument. Kaufman’s blog posting suggested that it was a good thing for “HCI [to] be considered a science because science was practically the only measurable form of progress in the 20th century”. Scientism faces us all, especially when it comes to gathering funding and establishing the legitimacy of our research work in HCI. In one view this could be seen as a cynical perspective on science (and perhaps devaluing the label), but on the other hand, we can instead see it as ‘strategic’ and necessary.

[An aside: for those who are not familiar with the CHI reviewing process, a simplified version is as follows. The conference is organised into various subcommittees which cater for different aspects of the field of HCI (this subdivison is of course a massive fudge, but practical). Authors target their paper to a given subcommittee; the subcommittee’s chairs then divvy up the papers between themselves and the associate chairs (ACs), who in turn recruit a number of reviewers (3+) for each of the papers they are AC-ing for. Reviewers do the reviewing, ACs write meta-reviews summing up the points made and scores provided by the reviewers, all of which are then sent to authors. Anonymity is top down and bottom up: authors are anonymised for reviewers, and the reviewers are anonymised for the authors. Reviewers are also anonymous between each other. ACs on the other hand know who the reviewers are. Authors, having received their paper’s reviews have the option of writing a rebuttal if they so wish. After the rebuttal period is over, reviewers can alter their review and score based on the rebuttal. The final set of reviews (sometimes augmented with further reviews for particularly divided reviewers) are then used to work out what papers get discussed at the subcommittee meeting. The ACs and chairs meet up, discuss all the papers they feel need discussing (a very opaque process but generally speaking papers below a certain average score are typically dismissed), and then collectively determine accept / reject decisions for all the papers for their particular subcommittee. Finalised reviews and final paper decisions, often including some account of ‘what went on’ at the subcommittee meeting, are then sent to authors, who are either left whooping with joy or crying in despair.]