the semblance of objectivity in numbers

I just received my first ever first-authored conference paper rejection from FSE. The primary reasons, quoted from the reviews, include:

• “The qualitative nature of the study … is liable to misinterpretation and bias.”
• “I was expecting a quantitative analysis: is there any correlation between some of the characteristics and between [the results] and the time a bug takes to resolve and its resolution status?
• “I would have thought that what types of elements to look for in discussion should be decided before by the researchers as it should be based on the problem”
• “I was expecting concrete advice on HOW the tools should structure the discussion.”

I was hoping the reviewers would have been more epistemologically informed. For example, the first and second quotes are quite telling: they imply that some forms of empiricism are not subject to misinterpretation or bias. But quantitative empirical measures are just as subject to bias as any other measure. For example, if I had counted certain kinds of data and run correlations between these counts and other outcome measures, not only would one in twenty of them be “statistically significant” by chance, but whether there was any real meaning in the variables depends on the construct validity of the quantitative measurements. For example, if I had correlated hyperboles with bug resolution time, not only would the hyperbole measure have the same limitations as it did as a qualitative classification, but the bug resolution time would have any number of contextual factors that could influence its true reflection of the hyperbole’s impact on consensus. Transforming empirical observations into numbers does NOT make them objective, nor does it prevent bias and misinterpretation.

The third quote is ironic: this reviewer seems to believe that the only way to analyze a problem is to make some assumption about its nature upfront. The whole point of qualitative research is that the more you make upfront assumptions, the more you bias your findings. What this reviewer is proposing would have lessened the objectivity of the results and prevented us from uncovering the trends we did.

The last quote reveals the systemic bias in software engineering research (and also some HCI venues): qualitative studies are only valuable if they explicitly inform design. What this really reduces to is a view that material goods are real work, but the production of knowledge comes for free. Building a system or automating some activity, even if the system and automation are entirely impractical in the real world, is more valuable than understanding the real world. The comment also reveals the reviewer’s lack of understanding about design: innovations don’t come from studies, they come from people. Studies can support design decisions (and the results throughout our rejected submission have been quite valuable in our current design efforts), but they cannot generate ideas. People generate ideas.

Had I really wanted the paper in, I would have littered the submission with arbitrary, but seemingly objective quantifications and correlations of our data (which is what most quantifications are in software engineering papers). This has worked in past papers and is a tried and true workaround for the software engineering community’s lack of experience with qualitative methods. Reviewers would have thought, “I don’t get all of this qualitative stuff, but these numbers are great.” I decided not to do this on principle, since doing so would have only made the results seem more objective without adding any real objectivity.

So much for principle. Time to start correlating things!