what’s surprising?

A common complaint of research is that it’s not “surprising.” For example, a reviewer might say, “The study was well done, but the results weren’t really that surprising.”, or, “I found the results a bit predictable.”

But what do these statements really mean? Do they mean, “Had you asked me the research question, I could have guessed the results with some degree of confidence.”? Or, “If you asked your research question of 100 experts, 95 of their guesses would have been right.”?

Maybe we might intend for them to mean that, but they don’t actually capture what happens when a reviewer reads a paper. What usually happens is the reviewer reads the research question and thinks, “Hm, I could guess, but I’m not sure.” Then, upon reading the results, the reviewer thinks, “Well of course, that’s not surprising at all.” The test executed here is not whether an expert can confidently predict the answer to a research question, but whether in hindsight it seems plausible that an expert could have guessed the result.

In this sense, what makes a result “surprising” has less to do with what we know as scientists and more to do with what we think we know about what other researchers know.

This social fabric that apparently underlies our judgements of what is known has other interesting effects on what is accepted as advancing knowledge. For example, that some finding has not been published, is rarely a satisfactory argument for why something should be published. What underlies this belief it that it is not our goal as scientists to document everything that we know. Instead, it is our job to document the subset of what we know that is interesting, important, and surprising.

But aren’t most judgements of what is interesting and important are grounded in the present? How are we to know what is interesting or important in the future? Who are we to judge that the future of humanity will find no interest in the uninteresting, unimportant results of today? Take, for example, a recent review I wrote on a paper about using multitouch, tabletop displays for engineering design. I argued that it was unclear what problem was being solved. But what if it solves a problem that doesn’t exist yet? Or what if it solves it in such a way that another problem I hadn’t even thought of becomes trivial? On what basis could I really judge whether the work would have future worth?

All of this makes me think I don’t give papers a fair shake. Maybe I’ll adopt a new reviewing protocol: instead of reading the paper straight through and recording my thoughts, I’ll look at the authors’ research question and try to answer it myself for five minutes. Then, I’ll read the paper and if they came up with a different solution or answer that mine (that is of course reliable, sound, etc.), whether or not I’m surprised, the authors get credit for discovering or inventing something that I didn’t know. Of course, If I guessed their results or solution in a mere five minutes, what could they possibly have contributed?