What different survey modes and question types can tell us about Americans’ views of China

(Pew Research Center illustration)

(Related post: Trends are a cornerstone of public opinion research. How do we continue to track changes in public opinion when there’s a shift in survey mode?)

Every year since 2002, Pew Research Center has polled people in the U.S. and around the world as part of a major, cross-national study known as the Global Attitudes Survey. This year, for the first time, we conducted the U.S. component of the survey through our online American Trends Panel (ATP) instead of by telephone. The change is part of a broader shift at the Center, which now conducts nearly all of its U.S. surveys using the ATP instead of phone polls.

As previously explained in this blog, a change in survey mode can present problems for researchers trying to interpret trends in public opinion over time. To detect potential mode effects ahead of transitioning the U.S. component of the Global Attitudes Survey from the telephone to an internet-based panel, we fielded concurrent surveys online and on the phone in 2020, asking the same questions in both formats. We discovered several notable differences in this experiment, including in the U.S. public’s views of China. For example, the share of Americans with an unfavorable view of China was 13 percentage points higher on the ATP than on the phone (79% vs. 66%). Potential reasons for this disparity include the absence of a “don’t know” category on the ATP (7% of phone respondents in 2020 answered “don’t know”) and that self-administered online surveys may be less prone than phone polls carried out by live interviewers to social desirability bias effects.

In this post, we’ll explore additional nuances we recently observed in Americans’ views of China and conclusions we feel comfortable drawing from our Global Attitudes Survey trends in the U.S., given the challenges of a shift in survey mode.

Background

Our U.S. phone surveys in recent years have revealed increasingly negative views of China, with the share of Americans with an unfavorable view rising 26 percentage points between 2018 and 2020. As we developed our 2021 survey, we wanted to find out whether this trend has continued, particularly in the context of COVID-19, tariffs, human rights violations and other important issues. To help answer that question, we designed the 2021 survey with two different questions about overall views of China, each one asked of half the sample.

First, we included a traditional, four-point question we have previously fielded on the phone: “Please tell me if you have a very favorable, somewhat favorable, somewhat unfavorable or very unfavorable opinion of China.” Given the differences we observed in our concurrent web and phone surveys in 2020, we decided we would not be able to “trend” this question; that is, we would not be able to look at change over time on this particular measure by comparing the 2021 ATP findings to past phone results.

Second, we included a question that examined Americans’ views of China using a different measure, one for which we do have directly comparable ATP data from 2016 and 2018. It reads: “We’d like to get your feelings toward some different countries in the world on a ‘feeling thermometer.’ A rating of zero degrees means you feel as cold and negative as possible. A rating of 100 degrees means you feel as warm and positive as possible. You would rate the country at 50 degrees if you don’t feel particularly positive or negative toward the country. How do you feel toward China?”

Regardless of mode or measure, most Americans have a negative view of China

As we examined the results for these two questions in our 2021 survey, a few things became clear.

First, on both questions, a majority of Americans had negative views of China: 76% of U.S. adults who responded to the standard four-point question in the online panel had an unfavorable view, while 67% of those who were asked the thermometer measure said they had a “cold” feeling (anywhere from 0 to 49 on the scale).

Second, Americans’ views of China have indeed become substantially more negative in recent years. Looking at the thermometer measure, the share of adults with “cold” feelings toward China rose from 46% in 2018 to 67% in the new survey. The share with “very cold” feelings (0 to 24 on the thermostat) roughly doubled from 23% to 47%. In our previous phone surveys, we observed a similar rise: In 2018, 47% expressed an unfavorable view of China, a share that climbed to 73% in 2020. Notably, in the ATP version of the four-point question — the only one for which we are able to make comparisons between 2020 and 2021 — there was little change in the share of Americans who said they had an unfavorable view of China (79% in 2020, 76% in 2021).

Third, demographic differences were similar across the two measures, suggesting that the general patterns we found are consistent regardless of how the question is asked. For example, on both the four-point question and the feeling thermometer, men were more likely than women to hold very negative views of China. The same was true for older Americans versus younger Americans as well as for Republicans and Republican-leaning independents versus Democrats and Democratic leaners.

The differences between the two measures related to the ‘neutral’ option

Still, there were some differences between the feeling thermometer and the four-point scale. For example, the share of Americans who expressed a “cold” feeling on the thermometer was lower than the share who said they had an “unfavorable” view on the four-point scale (67% vs. 76%), even as the share of Americans expressing “very cold” ratings was higher than the share expressing “very unfavorable” views (47% vs. 39%). In addition, roughly one-in-five Americans rated their feelings as 50 on the thermometer, essentially choosing a “neutral” midpoint that was not available to those who received the four-point question.

There are various reasons why our respondents might have chosen this neutral category. Some respondents may not have heard much about China and therefore didn’t feel informed enough to answer the question. For some, this may have been an issue of “satisficing” — that is, choosing an answer that is easier cognitively instead of engaging with the question, just to keep the interview moving along. It’s also of course possible that many respondents genuinely didn’t feel positively or negatively about China.

Indeed, the results of an open-ended question we asked on the same survey indicated a mix of all of these reasons. Among the responses given by those who chose the neutral option, 6% wrote something like “I don’t know much about China” or suggested that they hadn’t heard enough to write anything in detail. (This was distinct from simply skipping the question.) The other most common responses among this “neutral” group involved China’s economy (including the idea that everything is “made in China”), China’s relationship with the U.S. (including comments about how the U.S. economy is impacted by China), China’s politics and government, references to human rights abuses, mentions of COVID-19, the idea that China has a large population or is overpopulated, and pollution. (For more on the open-ended responses and the methodology, see “In their own words: What Americans think about China.”)

Notably, some demographic groups chose the neutral midpoint at higher rates than others. For example, women were more likely than men to say they felt neither warm nor cold toward China, as were Democrats compared with Republicans.

The pros and cons of each measure

There are some key differences between the feeling thermometer and the four-point scale. The thermometer is administered using text box entry and 101 answer options — any number from 0 to 100. We then assign meaning to the 101 values by grouping them into categories, treating 50 as a neutral “neither cold nor warm” group. The four-point scale, in contrast, involves a conventional display where respondents choose one of four visible options.

We often stay away from using scales with neutral midpoint options (unless the midpoint is itself a useful category) because “satisficing” is a well-documented problem in survey research. When asked the four-point question, which does not have a neutral middle category, most respondents pick a side rather than skip the question altogether. We can then also see the degree of these feelings — “very” or “somewhat” — which is valuable for understanding political and social attitudes. In some ways, these factors make our traditional four-point question preferable to the feeling thermometer.

The four-point scale also leads to easier analysis and reporting. There are fewer categories, and readers can more easily grasp what “favorable” and “unfavorable” mean. The thermometer, on the other hand, requires us to explain how terms relate to numerical codes (e.g., “cold” is 0 to 49). Additionally, Pew Research Center’s international survey work is often conducted on the phone. While we typically include a visual prompt to assist with the thermometer question in our online polls, we can’t do that over the phone, making the question more cumbersome for the respondent.

Still, the thermometer does have some advantages. Respondents are given the opportunity to pick the position closest to how they feel with greater precision. In our 2021 survey, for instance, 32% of Americans said their views of China fell below 10 on the 0 to 100 scale, including 24% who said their feeling was a zero out of 100. Density around an endpoint with this level of detail is something we don’t see on a four-point scale. Additionally, researchers have several choices of how they can analyze the data. We focused here on frequencies, but one could instead look at mean ratings. We concentrated most of our analysis on the 0–25 “very cold” feelings discussed in our report, but researchers could also examine those who choose 0–9 or zero itself, offering a greater degree of flexibility.

Conclusion

We see that across modes and measures a majority of Americans have negative views of China, and that unfavorable views have increased in recent years. In addition, thanks to simultaneously asking a feeling thermometer question, we gained additional insight into the share of Americans who may not feel particularly warm or cold toward China as well as the intensity of the animosity some Americans feel.

Given the challenge of effectively and efficiently deploying a feeling thermometer question by phone, we are unlikely to abandon our four-point measure of attitudes toward China anytime soon. But we now have knowledge about split samples and alternate measures that can aid the Center as we contemplate similar mode transitions in other countries.

This post was written by J.J. Moncus and Laura Silver of Pew Research Center. J.J. Moncus is a research assistant and Laura Silver is a senior researcher. Both focus on global research.