Incivility in Open Source Code Review Discussions: A Characterization Study

This blog post summarizes our work entitled “The “Shut the f**k up” Phenomenon: Characterizing Incivility in Open Source Code Review Discussions” by Isabella Ferreira, Jinghui Cheng, and Bram Adams presented at CSCW’21.

The concept of incivility, i.e., heated discussions that involve personal attacks and unnecessary disrespectful tone, was traditionally focused on political and religious debates. Nowadays, however, with the increasing opportunities for online public debates, incivility has received increasing attention in many fields, including the field of software development. In this project, we directly tackle this concept, aiming to characterize incivility in the context of open source software development in general and code review discussions in particular.

Code review is a software quality assurance practice in which someone other than the author of the changed source code inspects the changes, provides feedback, and decides if the changes should be applied to the code base. The existing knowledge about incivility in this context is very limited. Many studies have investigated the technical aspects of code review, but only a few of them focus on the social aspects, often not directly targeting incivility. Thus, many questions remain to be answered, such as: How much incivility exists in code review discussions? Which moods or styles of expression in code review discussions demonstrate incivility features? What are the causes and consequences of incivility in the context of code review?

We focused on the Linux kernel development community to answer these questions. Particularly, we analyzed a representative sample of 1,545 emails that contained code review discussions associated with rejected patches (code changes not integrated into the Linux kernel) from the Linux Kernel Mailing List (LKML), the central space for discussions about Linux kernel development. We derived the following results.

Bitter frustration, name calling, and impatience are the most frequent features of discussion in uncivil emails

Through an inductive analysis (an approach to derive codes from the data), we identified features of discussion that were grouped into positive, neutral, negative, and uncivil (i.e., unnecessary disrespectful tone) categories (see Figure 1). The analysis was first conducted at the sentence level, then the results were aggregated in the email and thread levels (an email thread includes all emails replying to the same topic).

Figure 1: Frequency of features of discussion in code review discussions of rejected patches. Note: A sentence can be coded with multiple codes.

We found that humility is the most frequent positive feature, and that the most common uncivil features are bitter frustration, name calling, and impatience.

Two-thirds (66.66%) of the non-technical emails are uncivil

Based on the features of discussions, we classified emails into three groups: technical, civil, and uncivil. Technical emails are those that contain only technical discussions and do not contain any feature of discussion listed in Figure 1. Civil emails are those that express positive, neutral, and/or negative features of discussion. If an email expresses at least one uncivil feature of discussion, the email is then classified as uncivil.

We found that 1,377 (89.13%) of the analyzed emails are technical. This is expected because of the technical focus of LKML. However, 112 (7.25%) emails were identified as uncivil and 56 (3.62%) as civil. This result is surprising since it shows that more than half of the non-technical emails (out of 10,87%) are uncivil (66.66%).

Causes and consequences of uncivil communication

We analyzed the content of the email that the uncivil email is replying to in order to assess what triggered the incivility in practice, and we focused on the email that replied to the uncivil emails to assess their consequences. Figure 2 presents the relationship of the causes and consequences of uncivil emails sent by developers (those who proposed code changes for review) and maintainers (core community members responsible for reviewing the code changes).

In uncivil emails sent by developers, we observed that

• maintainers often discontinued the discussion when developers sent uncivil emails due to violation of community conventions (e.g. workflow imposed by the community) or when the maintainer’s feedback proposed a non-optimal solution;
• maintainers often escalated the uncivil communication when the cause of incivility was the maintainer’s feedback, communication breakdown (e.g. misinterpretation), maintainer’s behavior, or when developers performed an inappropriate solution;
• maintainers often reinforced their standpoint when developers were uncivil due to the maintainers’ uncivil behavior.

When maintainers sent uncivil emails, we found that

• developers often accepted the maintainers’ criticism and addressed the changes suggested by the maintainer, discontinued further conversation with the uncivil maintainer, or discussed the problem in a civil way;
• developers escalated the uncivil communication when the cause was communication breakdown;
• developers usually called out the uncivil behavior because the behavior was caused by a violation of community conventions.

Figure 2: Relationship between causes and consequences of uncivil emails sent by developers (left) and maintainers (right) when discussing rejected patches.

Overall, our work provides empirical evidence that incivility is not neglectable in code review discussions of rejected patches. Based on the uncivil features of discussion found in our work, if code review discussion participants avoid the expression of bitter frustration, name calling, and impatience, reviews and arguments tend to be more constructive and efficient. We also suggest practical implications and tool design ideas to prevent and address incivility. Particularly, tools that detect the different uncivil features of discussions and/or address the various causes of incivility could support code review discussions.

For more details on this study, please read our paper. For any questions, please feel free to reach out to the authors.

Isabella Ferreira, Jinghui Cheng, and Bram Adams. The “Shut the f**k up” Phenomenon: Characterizing Incivility in Open Source Code Review Discussions. 2021. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 353. (October 2021), 35 pages. https://doi.org/10.1145/3479497

Finally, we are currently conducting interviews with FLOSS contributors that have experienced/witnessed disrespectful conversations. If you are interested to participate in our interview, please fill this short form and we will contact you to schedule an interview: https://bit.ly/3xE2SqV