reflections on conference papers and journals
For the first time in my academic career this week, I was working on a journal paper and a conference paper at the same time. This wasn’t entirely intentional; both of these papers were going to be CHI papers, but as the results and writing for one of them materialized, it became clear that not only was the audience not a fit, but I actually couldn’t fit all of the important results into 10 page SIGCHI format. This realization, and the fact that I was working on both simultaneously, led several realizations about how the two kinds of submissions differ.
First and foremost, the lack of a strict length restriction on the journal paper was surprisingly freeing. While on the CHI paper, every other discussion with my student was what to cut and what to keep, discussions about the journal paper were much more about what details and results were missing. Obviously, there are advantages to each: with the CHI paper we were probably forced to be much more concise and selective about the most significant results; similarly, the journal paper was slightly more verbose than it needed to be, because I didn’t have the threat of desk rejection to force more careful editing. At the same time, there were many interesting things that we had to leave out of the CHI paper that could have fit into just one additional page. With journal paper, the question was not “what’s most significant?” but “is this complete?”
The length differences also had a significant effect on how much space we gave to details necessary for reproducibility. For the journal paper, I felt like our task was to enable other researchers understand exactly what we did and how we did it. With the CHI paper, our task was to provide enough detail for reviewers to see the rigor of what we did, but the amount of detail we ended up including really wasn’t enough to actually reproduce our study. In the long term, this is not good science.
Although the journal paper didn’t have a deadline, I did impose one on my lab in order to align with the end of summer, since the undergrad research assistants on the paper would have to resume classes (as would I). The deadline worked well enough to motivate us to finish the paper, but it also freed us to take an extra day or two to read through the manuscript a few extra times, improve some of the figures, and verify some of the results that we felt may have been done too hastily. The CHI paper, in contrast, was rushed, as most CHI submissions are. There was just enough time to edit thoroughly yesterday and submit today, but there’s an extensive list of to do’s that we have if the paper is accepted. Sure, we could do them now, but why not wait until reviewers provide more feedback? With the journal paper, we submitted when we felt it was ready.
Of course, the biggest difference between the two submissions has yet to come. In November, we’ll get CHI reviews back and likely know with some certainty whether the paper will be accepted or rejected. There will be no major revisions, no guidance from reviewers about what they’d like to see added or changed, and certainly no opportunity for major improvements if it is accepted. Instead, the reviews will focus on articulating a rationale for acceptance or rejection. With the journal paper, I’ll (hopefully) get three extensive positions in a few months on what is missing or wrong with the paper and what they’d like me to change in a revision. The process will likely take longer, but in trade, I hope the paper will be much better than original manuscript.
One of these processes is designed for speed, the other is designed for quality. I’ll let you guess which is which. And let me be clear: I’m a big fan of conferences. Most of my work is published at major HCI and software engineering venues and not journals and I truly enjoy the fact that nearly everyone in our community rallies together at the same time of year to contribute our latest and greatest for review. But as someone who has the freedom to really publish in either, I’m really starting to question whether the average conference paper can actually be of comparable quality to the average journal paper. There might just be inherent limits to a review process that is optimized for selecting papers for presentation rather than improving them.
Of course, this isn’t a necessary dichotomy. I’ve talked to many people in my research community about blending the two. For example, if we simply had journals of infinite capacity and no conference papers, and instead put all of our reviewing effort into our journals, we could easily design an annual conference where people present the best work from recent journal publications (and work in progress, as we already do). In fact, CHI already lets ToCHI authors present their recently published papers, so we’re part way there. With changes like this, we might find a nice balance between a review process designed for improving papers and a conference designed for fostering discussion about them.
