Evaluating changes in how we ask Americans for their Twitter profiles
Starting in 2019 and again in 2021, Pew Research Center has asked members of our nationally representative online survey panel, the American Trends Panel, if they use Twitter and, if so, whether they would be willing to provide us with their Twitter handles for research purposes. This has allowed us to study how U.S. adult Twitter users’ stated attitudes and opinions on a variety of topics correspond to their actual behaviors on the platform.
As with any survey question, the wording of an “ask” like this can affect how it is perceived and understood by the respondents taking the survey. In our March 2021 survey, we decided to take a fresh look at the consent language we used when asking Americans to give us their Twitter handles. Below is the original question wording.
We would like to better understand the role of Twitter in society; for example, how often people use Twitter and what they tweet about. To help us with this research, we hope you will share your Twitter handle with us. Your handle is the username you have selected for your Twitter account (like @yourhandle). We will treat your handle with the same care we take with all of your survey responses — it will be used only for research purposes, and we will never share any of your tweets or any profile data that can be linked back to you. We expect that this research will be concluded within 18 months.
Please list your Twitter handle in the box below.
[MEDIUM TEXT BOX WITH @ IN FRONT OF THE TEXT BOX]
[Twitter handles in box: 1) cannot contain ‘Twitter’ or ‘Admin’, 2) Cannot be longer than 15 characters, 3) only alphanumeric characters (A-Z, 0–9) and underscores are allowed. Capitalization doesn’t matter.]
SHOW ON SEPARATE SCREEN IF TWITTER HANDLE GIVEN (TWITTER2=VALID TWITTER HANDLE ENTERED):
If at any point you wish to opt out of research related to your Twitter account, contact info@americantrendspanel.org. For more information about our privacy practices, please see our Privacy Policy.
The original language we used in our 2019 survey had the benefit of being simple and readable — perhaps overly so. It wasn’t entirely clear, for instance, what information we would actually be getting from people’s Twitter accounts.
When we revised our question wording in 2021, we first wanted to give more details to our panelists before they consented. That meant describing as clearly as possible — within the constraints of our existing survey design practices — what information we would and would not get from them, what we would do with that information and how long we would continue our research project. We also wanted to make sure we conveyed all of this information without using unnecessary technical jargon and that we kept the text at a relatively simple reading level.
In addition to these changes, we tried using a “loss framing” to describe why we wanted respondents’ Twitter handles. That is, rather than describing what we would be able to do if they gave us their information — as we had done in our first effort — we described the research we would not be able to conduct unless they gave us a handle. This change was motivated by evidence that emphasizing negative effects of nonconsent is a more effective motivator for survey respondents than emphasizing the benefits of consenting, at least when it comes to asking for permission to link survey responses to external data.
The updated question wording, with changes and additions in bold text, is here:
In many of our surveys, we ask about how people like you use social media. But there’s a lot we cannot learn unless you allow us to connect information about your public Twitter behavior to your answers.
No matter how often you use Twitter, would you be willing to share your Twitter handle to help us with this research?
1 Yes
2 No
This project will ONLY collect information that an ordinary user would see when they visit your profile (see here [POPUP] for more details). We will treat this data just like we treat your survey responses and will NEVER share anything that can be linked back to you.
[POPUP]
How we will use your Twitter handle
If you provide us with your Twitter handle, we will only be able to collect what an ordinary user would see when visiting your profile. The specific information we will have access to depends on the privacy settings of your Twitter account.
If your account is set to PUBLIC, we will be able to see the accounts you follow; the accounts that follow you; the text of any tweets you have posted; the basic information in your profile, such as your description and the age of your account. We will NOT have access to any private messages you have sent or received.
If your account is set to PRIVATE, we will be able to see how many accounts you follow or how many times you have tweeted — but will NOT be able to see any private messages you have sent or received; the actual content of your tweets; or the names of the accounts you follow or that follow you.
We expect that this research will be concluded in approximately 18 months. At that point we will ask you again whether or not you would like to provide your handle. If you provided your Twitter handle to us previously, that is why you are seeing this question again.
If at any point you wish to opt out of research related to your Twitter account, contact info@americantrendspanel.org. For more information about our privacy practices, please see our Privacy Policy.
[END POPUP]
Please list your Twitter handle in the box below. Your handle is the username you selected for your Twitter account. It is no more than 15 characters long and looks like @yourhandle.
[MEDIUM TEXT BOX WITH @ IN FRONT OF THE TEXT BOX]
[PROGRAMMING NOTE: Twitter handles in box: 1) cannot contain ‘Twitter’ or ‘Admin’, 2) Cannot be longer than 15 characters (do not display character limit on screen), 3) only alphanumeric characters (A-Z, 0–9) and underscores are allowed. Capitalization doesn’t matter.]
Text for prompt when invalid response is entered: Answer is not valid. Your handle can only include alphanumeric characters (A-Z and 0–9) and underscores.
Assessing the results of our revised question wording
So how did response patterns change after we revised our question wording?
In some areas, we saw improvements. In our 2019 survey, for example, around one-in-ten volunteered Twitter handles (11%) were deemed invalid because the account did not exist, the basic demographics of the account obviously contradicted their survey responses, or for some other reason. The percentage of invalid handles was only 6% in 2021.
But while a slightly larger share of handles were valid this time around, a lower overall share of Twitter users were willing to provide us with their handles in the first place. In our 2019 survey, 21% of our panelists reported being Twitter users and 59% of them provided a handle of any type. In our 2021 survey, a similar share (22%) reported being Twitter users — but just 43% of them provided us with a handle.
After accounting for the differences in handle quality noted above, the total share of Twitter users who provided a valid handle was 53% in 2019 and 40% in 2021.
What explains the lower consent rate?
Increased participation was not the primary reason we changed our consent language in 2021. Still, we had hoped that our adjustments might encourage more respondents to offer us their handles. So why did the opposite occur?
Unfortunately, due to sample size limitations, we didn’t have the option of experimentally testing different versions of the consent question. We could only evaluate one question fielded in 2019 and a different one fielded in 2021. As a result, we couldn’t really tell how much of the decline in the consent rate was due to the question itself, to the changing composition of the American Trends Panel (which regularly gets “refreshed”) or to other external factors.
To try to shed more light on this question, we looked at the 2,644 Twitter users in our 2021 survey and examined whether they participated in the 2019 survey, too. By focusing on the individuals who responded to both surveys, we thought we might be able to discover factors other than question wording that may have influenced their decision to consent.
We learned a few things from this analysis. For one, people who received the question for a second time tended to respond the same way they did the first time around. It also turned out that we were able to convince a sizable share of panelists who were unwilling to share their Twitter handles in 2019 to do so in 2021–16% of them, to be exact. That group, however, was offset by “losses” from people who did give us a handle the first time around but declined to do so in 2021. Around 35% of this cohort declined to provide us a handle in 2021 after doing so two years earlier.
We also found that respondents who saw the consent question for the first time in 2021 — either because they were new to the American Trends Panel or because they were existing panelists who were not Twitter users in 2019 — consented at lower rates than we saw in our first effort.
To investigate further, we decided to look more closely at respondents who received the question in both 2019 and 2021 to try to find other patterns. First, we divided those respondents into two groups: those who consented in 2019 and those who declined in 2019. We then looked to see if there were any demographic or behavioral factors that were associated with agreeing to provide a Twitter handle in 2021.
Overall, there were few substantive differences between these groups by race, gender, age or party identification. The significant differences that did exist often told unclear or even contradictory stories. In the case of privacy attitudes, for instance, panelists who provided us with Twitter handles in 2019 were more likely to consent again in 2021 if they believed that only a few people see their tweets. However, this group was also more likely to consent again if they believed that it would be easy to identify their social or political views based on the content of their Twitter account or said that they use Twitter primarily to express their own opinions rather than see the views of others.
Similarly, panelists who did not provide a handle in 2019 were less likely to consent in 2021 if they viewed harassment on Twitter as a problem. But for respondents who did give a handle in 2019, there was no relation between their views of harassment on Twitter and their willingness to provide a handle in the subsequent ask.
In the end, it’s a bit of a mystery why our changes in question wording resulted in a lower consent rate in 2021.
Takeaways
Every respondent in a survey is important, and it’s always unfortunate to “lose” someone. That’s especially true when you’re recruiting from a relatively small pool to begin with, as is the case with Twitter users.
At the same time, we know the Twitter users who provided us with a handle in 2021 are still largely representative of Twitter users more broadly — especially after we weight the sample appropriately, as we discuss in this companion post. The 2021 sample also remained large enough to analyze major subgroups of adult Twitter users in the U.S., such as men and women or Democrats and Republicans.
There are also benefits to using our new consent language that go beyond raw numbers. Most importantly, we can feel confident that our panelists have access to better information about what we are planning to do with their Twitter data and how long we will use their personal information. For this reason alone, we can consider this adjustment a success, even if we didn’t reach our loftiest ambitions in terms of convincing more respondents to provide us with their handles.
This post was written by Athena Chapekis and Emma Remy, who are data science analysts at Pew Research Center.
