User Testing Report

Methodology

Overview

User testing was conducted to test the perception of value of the Phase 1 design of the browser extension product, Keeper.

Keeper is a browser extension for Facebook that facilitates more civility and empathy in Facebook conversations. Through Facebook Community Standards annotations, personal value entry and display, and an alternate conversation view, Keeper provides opportunities on Facebook to surface the things that connect us as humans, beyond our social network.

The goal of this research was to answer the following questions:

• How likely are users to use this product?
• Do users understand the overall benefit of the product?
• What are the risks associated with the product?

To accomplish these goals, users engaged with a four prototype-stage features of the Phase 1 design. Users were prompted to imagine scenarios when they would use the product features, and shared their immediate reactions, perceptions of value and likelihood of use.

Recruiting

• Participants were recruited via Craigslist
• Each participant received a $25 gift certificate compensation
• 8 moderated, recorded remote interviews took place via Zoom
• Moderated by user researcher/SVA IxD graduate student
• Held on Saturday, March 16 and Sunday, March 17

User Demographics:

• Mix of gender
• Mix of age groups (15–24, 25–39, 40–65)
• User Behaviors: Has commented on content from the Facebook pages of brands, businesses, news media, groups (closed or public), public figures or individuals within the last 6 months. Mix of frequent, occasional, and rare comment exchanges with Facebook users to whom they are not connected on the platform and whom they do not know offline. Frequent = reads comment threads and comments on a post at least once a week. Occasional = reads comment threads and comments on a post at least once a month. Rare = reads comment threads but rarely or never comments on a post.

Observations + Insights

Feature 1: Annotated Community Standards

Observations
Most users said they would use this feature. However, they had different perspectives on the problem it was trying to address. One user said they would use it because they could imagine the feature helping to interrupt the “stream of information” they’re consuming on the platform, and cause them to pause and think about how they are behaving and what they are doing.

Two users associated this feature with Facebook’s enforcement of their community standards, which they felt was inconsistent. One of them said they would still use it, the other said they would not. Both felt that they, as individuals, had a good sense of what the community standards contained and how to conduct themselves in a safe and respectful way on Facebook, but did not feel that Facebook was uniformly enforcing the standards. The user that said they would use the feature said that despite their concerns about Facebook enforcing the standards, having them visible and easily acceptable helped her think of Facebook as a whole community and helped her think of the guidelines with more objectivity.

The user who said they would not use this feature explained that they questioned the value of the community standards themselves, wondering what purpose they serve if they are not enforced. They felt that Facebook should be solving their own problems, and this feature was a waste of time for the non-Facebook team who was building it.

Two users suggested adding information or functionality about how to report things you see that violate the standards. One noted that one of the reasons they feel Facebook doesn’t do a good job of enforcing the standards is that the platform seems to prioritize flagrant violations, rather than “small ones” that only get a small number of reports. Only the flagrant violations (such as live-streaming a mass murder, as happened the day before most of these interviews) get attention.

Finally, two users suggested that this feature would be helpful for young people, such as students, who are just learning about the internet and how to interact with it.

Insights
I was surprised that two users immediately started talking about the enforcement of Facebook’s community standards (or lack thereof), and that this weakened the value of the Annotated Community Standards feature for them. Facebook has very publicly struggled to corral the behavior of its billions of users, many of whom have manipulated the platform to spread misinformation and incite violence, an outcome Facebook seemed utterly unprepared for. It seemed that from the perspective of at least two of the users I spoke to, Facebook should be responsible for what happens on their platform, and they are uninterested in third party products that would aim to make their experience on the platform better.

Feature 2: Values Wall

Observations
Only 29% of users said they would use this feature. The same amount said they might use it, and 43% said they would not use it. Several responded that it seemed “fine” or “nice” but that they didn’t fully understand what purpose it served.

Two users in particular said they liked it in theory, but felt that the concept of ‘values’ was much too nuanced to only represent it through single words. One explained that they felt like there are values that everyone knows they are supposed to have, and those would probably be the most popular. But, even if they state that they follow those values, they’re not always living up to them every minute of every day. For this user, this made the Values Wall feel disingenuous. They felt that seeing more context, such as the reasoning behind which values users choose to display, would make this feature more interesting. This user also pointed out that the values worked the same as any other information you might put on your profile — it’s self-reported by the user, so it’s telling the story the user wants the world to know. It’s curated as much as photos and posts. Additionally, this user had concerns about selection bias, as long as the creators of Keeper are choosing the values that users can display.

Insights
In general, the Values Wall was met with a lot of skepticism. In retrospect, I’m not sure this feature should have stood on its own in a testing environment. But my goal was to introduce it as a concept at this point in the interview, so that it made sense when the values appeared again in the Conversation View. I was also trying to gauge whether this feature came off as underdeveloped or inauthentic, which it did.

For the most part, the users I spoke to had a good sense of the ways in which Facebook, and the internet in general, could skew their perceptions of each other. They generally accepted as “part of the deal” of using Facebook that the picture people paint of themselves online isn’t always reflective of reality. In light of this, the Values Wall, while a slightly different kind of information, felt just as curated as everything else, and didn’t impact anyone’s perception of the user whose values they were looking at.

I do think the concern about selection bias is fair and brings up an important risk. I had not explored how my product would be able to create a broad, comprehensive list of values, but it has certainly been a challenge looming in the distance. I intentionally decided that users would choose values from a list, rather than being able to write their own, in order to prevent users from listing values that contained hate speech or objective violations of the community standards. How can a service, such as Keeper or Facebook itself, avoid bias while also maintaining and enforcing objective standards for speech and behavior?

Feature 3: Conversation View and Values

Observations
Like the Values Wall, only 29% of users said they would use this feature. The same amount said they might use it, and 43% said they would not use it. Several seemed intrigued by the design of the alternate conversation view, but uncertain if it would improve the comment reading experience. Two users felt that hovering over avatars to read comments in the Conversation View seemed more time consuming than how they currently read comments, by scrolling through them.

In viewing the abbreviated profile view of a commenter in the conversation view, which includes just their name, values from their Values Wall, and their comment on that particular post, the users had mixed reactions. One user was adamant that this feature provided no additional value and it just felt like a “novelty filter.” Their reasoning was that, in their opinion, the existing comment interface is fine, and they do not believe it was necessary to change anything about it.

This user, along with two other users, pointed out that if the visible comment did not align with the values the user had selected for display, then they might think less of that user, rather than more. This would render the values meaningless. If the comment was offensive to another user, the values might come across as disingenuous or boastful, and might encourage that offended user to respond with equally harmful language, as opposed to lessening the offense taken.

Two users suggested finding ways to make the values that are displayed have some kind of connection to the comment itself, either through machine learning or through self selection by the user at the time of commenting, e.g. answering the prompt, what values does this article/comment represent for you? One of these users thought it would be interesting if they could see what other comments other users have left, as a way of learning what kind of language this person typically uses and in what ways, and what other articles and topic areas they engage with.

Two users also pointed out that the values could provide another piece of information to “pick at”. In other words, another thing someone could use against you if they are attacking you. They might say, “Oh family is your value? Well I think you’re a terrible mother!”.

Insights
The users I spoke to surfaced two really big risks that I hadn’t considered. First, that in an instance of a user’s values not aligning with their comment, other users could have a negative perception of that person. This is the opposite of what I was hoping the design would do. It’s possible that developing the nuance of the values, i.e. expanding them beyond just individual words, could mitigate this risk — if the nuance was strong enough to conjure enough empathy and connection that users would pause before making a knee jerk reaction or judgment. But what could that design look like? And would it be possible for one design to have the same effect on every single user? Perhaps one of the challenges of managing so many users on one platform is exactly that — the variety of contexts, cultures and learned behaviors is just too vast for a one-size fits all approach.

The second risk was that the values could become just another reason to attack or criticize other users. This really highlighted for me that users don’t see values as anything different from the information they already put on display about themselves. Even though we are all likely to hold similar values as guiding principles in our lives, even across generations and cultures, they do not make up for the absence of the conversational queues we rely on in person to connect and understand each other, the things that do actually make us all human.

Feature 4: Comment Review

Observations
Most of the users I spoke to (67%) stated that they would be likely to use the Comment Review feature. 33% said they might use it, while no one said they would not use it. One person was not tested on this feature due to time constraints.

Two users wanted to see different instructional language in the review prompt window. One commented that it felt too easy to dismiss, because the only answer options to the question, “Are you sure you want to post this?” are “Yes” and “Edit.” “Yes” is first, so it could be easy to speed right through the interaction. The other suggested more direct language that clearly stated what the prompt wanted to prevent you from doing, e.g. “Are you sure you want to post this, because the language you’re using might be harmful to others.”

In general, users felt they personally had a good sense of how to communicate with others on Facebook in safe and respectful ways — so this feature would not fulfill an immediate need. Still, several felt this would be useful for others who typically do not have that sense, such as young people who are using Facebook for the first time.

Insights
I was surprised by the overall positive reception of this feature. Even if users weren’t certain they personally had a need for it, they clearly expressed that they could think of someone who would. One of these users also said that they know a second look at their writing never hurts, which to me implies they might be more willing to use this feature if it could serve as a subtle reminder to be thoughtful in their words.

One user also made an interesting observation about the audience for the product as a whole. They explained that after seeing the Comment Review feature, they understood better what Keeper was trying to do — in their words, encourage more productive discourse. If that was indeed the goal of the product, they felt that based on their understanding of the motivations and intentions of “trolls” and the like, the users who need this most are unlikely to acknowledge that need. The audience for this product is not trolls, but instead people who are already “desiring to participate in a thoughtful way.” This user encouraged Keeper to be transparent about that and to not hold back from making its intentions clear.

I’ve felt a lot of tension around this question. Anecdotally, I’ve found that users who are desiring of thoughtful discourse, generally don’t need help with it. I agree that the users who do need help with it (with the exception of students or other young people), would not have any interest in a product to fix it. So what would motivate them to use this feature? The user mentioned above suggested using it could serve as a way to pronounce themselves as someone committed to improving discourse on Facebook, someone committed to a cause. Perhaps there could be a way for users to show on their profile, with a badge or a profile photo frame, that they are using this product.

Implications + Next Steps

Implications

My user testing this week called two of four features into question, the Values Wall and Conversation View, as far as their perception of value. It also revealed to me that these features, especially the Values Wall, feel underdeveloped. Users expressed intrigue, but couldn’t immediately recognize what value the features could bring. I don’t know if that means that there’s nothing worth developing further, or if they were good ideas but weren’t developed enough at the time of testing.

I feel like I’m at a little bit of an impasse at this point. On the one hand, the users I spoke to had some good ideas for how to add more nuance to these to features and make them more compelling. On the other hand, while I would like to continue iterating on them, my timeline is a lot shorter than it used to be, and I’ve been struggling a lot lately with wondering whether I’m where I should be in the thesis process. I’m aiming for narrow and highly-developed as opposed to broad and underdeveloped.

The feedback to both the Annotated Community Standards and Comment Review features was solid enough that I feel comfortable saying my users found them to be valuable. One user noted that the Comment Review feature felt like it shifted the onus onto the user to take accountability for their actions, which is very much in line with my vision for this project. I think this feature could go along well with the Annotated Community Standards, which in its own way throws the accountability back on the user; this feature aims to make it so that there would be no denying that the community standards information was there and easy to find. Additionally, these two features don’t have to be used only on Facebook. They could be easily scaled for other social media platforms like Twitter or even Instagram. One of my concerns about all of this though is whether they feel enough like one packaged product, as opposed to two separate, independent products.

Drawing again from Nina Lysbakken’s work, the internet and especially social media came with so few guardrails for how to use them, unlike most other spaces we as humans travel through. Sometimes those spaces have guardrails established by governments and laws, sometimes by cultural traditions or etiquette, sometimes by the community of people who inhabit the space. In many ways, these guardrails have been successfully transferred to digital spaces. At the same time, the anonymity offered by the internet has allowed many of those rules to be broken, with few consequences. My product, with the two features described above, would aim to be the guardrails that Facebook hasn’t been able to provide. Since I like metaphors, it would almost be more like training wheels, in the sense that it’s not necessarily dictating what users should do, but it’s guiding them towards making as objectively safe and thoughtful decisions as possible.

Next Steps

My next steps will be in service of developing my Looks Like Feels Like prototype, per the syllabus: “This means everything important in your project should be fully visually designed. Some aspects of functionality can be fake/simulated, as a working version is not due until the final version.”

I will focus on developing and visually designing the two features, Annotated Community Standards and Comment Review. I will also work on developing the language used in both of these features, as a way to tie them together and make sure they are both in service of the same goal. I have not yet met with my thesis advisor this week, so this plan may change after Thursday morning. I will update this post with any additional information.