Introducing the GOV-BOTs psychometric scale for measuring users’ perceptions of bots in online communities

Although communities historically consisted of only humans, bots have now become influential social actors in our online communities. Indeed, governance bots do all sorts of fascinating, useful, and entertaining things across Reddit, and the number of active bots continues to rise as subreddits grow larger and more active.

Governance bots can help to automatically moderate posts and comments, share community rules, show up on command to perform silly functions, edit or convert content, send direct messages to users, and so much more. In our new paper, “The Impact of Governance Bots on Sense of Virtual Community: Development and Validation of the GOV-BOTs Scale,” we ask, how are such bots impacting our psychological experiences of virtual community?

To date, no prior psychological scales exist that enable researchers to measure users’ perceptions of bots in an empirically validated way. Therefore, our first task was to develop a new scale — we named it GOVernance Bots in Online communiTies (GOV-BOTs).

We began by collecting existing, rigorously validated instruments for measuring important community-related constructs. We build on validated scales in organizational psychology for measuring Sense of Virtual Community (SOVC) itself, as well as antecedents to SOVC and outcomes of SOVC. For example, Similarity and Interactivity are two constructs that have their own scales, have been well-studied, and have been repeatedly shown to lead to the formation of SOVC. Users’ Identification with the Community (along with many other constructs) has repeatedly been demonstrated to be present as an outcome, when SOVC is present.

We knew from prior research that users can perceive bots to have either beneficial or detrimental impacts on communities, therefore we hypothesized that users’ perceptions of them would impact their experienced SOVC, as well. That is, perceptions of bots should function as psychological antecedents to SOVC. Through a rigorous and exhaustive scale development process involving two phases of surveys deployed across 13 subreddits, we developed the GOV-BOTs scale to test this idea.

The GOV-BOTs scale includes two subscales to measure user perceptions of: (1) bot governance (4 items) & (2) bot tensions (3 items). Reddit moderators, researchers, and bot developers can now use this scale to empirically measure how users view bots as impacting their communities. The text of these items is included below, and can be measured with standard Likert ratings.

Second, using structural equation modeling, we show that effective Bot Governance does contribute to improving users’ sense of virtual community. This is important! When bots are present as social actors, they augment and contribute to governance efforts, with meaningful psychological impacts on users. Moreover, because we also measured well-known antecedents and one outcome of SOVC, and because those constructs all behaved in theoretically expected ways, we have a lot of confidence that our data and selected methods are sound.

On the other hand, perceived Bot Tensions do not have significant effects in the main model. But when we distinguish between users who indicate between being more or less aware of bot activities in the subreddits, bot tensions do have significant effects, but only when users are more highly aware of their activities, and those impacts are still positive. We discuss this interesting tension in the paper.

We also asked users, “Where do you personally experience the greatest sense of virtual community on Reddit?” and “To what extent do you feel that a sense of community is important to your experiences…” across a variety of social media sites. As you can see in (a) below, most Reddit users experience the greatest sense of virtual community across *groups* of subreddits, rather than individual subs, and (b) they view Reddit as a platform where SOVC is most important.

If we were to ask (b) to users of other platforms, we suspect those numbers might shift in favor of the platform wherever we ask the question. But it’s nonetheless interesting to observe that people perceive significant differences in their community-like experiences across today’s social media patterns. Taken together, these results strongly suggest that user-centered research should become more precise about how it uses the term “community” to describe these spaces.

Watch a short talk about this work at https://youtu.be/FycU8F-cTwQ and grab the full paper at https://bit.ly/govbots. The GOV-BOTs scale is ready and available for use out-of-the-box on Reddit, and can readily be adapted to many other platforms with bots, such as Discord, Twitch, Twitter, and more. We are excited to see how this tool will support participatory online community research and improve reporting on studies about bots in online communities. Thanks for reading, and enjoy #CSCW2022!