On a scale from 1 to 10, how much do the numbers used in survey scales really matter?
(Related post: Small changes in survey scales can matter when measuring political ideology in Europe)
You’re probably familiar with surveys that ask you to rate yourself on a scale from 1 to 10, whether they ask about your happiness, job satisfaction or even political leanings. Sometimes, these questions use a 0–10 scale instead. But do these numbers really matter, apart from their rank order? Is it possible, for instance, that some survey respondents might be more likely to describe themselves with the number 1 than the number 0 because 1 is a positive number and 0 is not?
Understanding how the numeric labels on scales might influence survey responses is an area of ongoing investigation for researchers. At Pew Research Center, we’ve spent some time thinking about this, too.
We recently ran an experiment testing whether the numeric labels used in one of our scales influenced how respondents reported their political ideology. We compared the results from a survey question that used 0 as the minimum value on a scale with one that used 1 instead. (Other researchers have considered how people respond to political ideology questions and the limitations of asking about political ideology more generally.)
We conducted this experiment as part of a 2017 telephone survey about religion and pluralism in Western Europe. (For the survey methodology, see here.) To carry it out, we randomly assigned respondents in France, Germany and the United Kingdom our political ideology question with one of two 7-point scales: either 0–6 or 1–7. The full question wording using our traditional 0–6 scale was this: “Some people talk about politics in terms of left, center and right. On a left-right scale from 0 to 6, with 0 indicating extreme left and 6 indicating extreme right, where would you place yourself?”
Our goal was to address several questions:
1. Do different scale endpoints result in different frequency distributions across the ideological spectrum or in different rates of item nonresponse?
2. Do differently numbered scales affect the broader left-right lean of the scale?
3. Do the two scales result in different demographic profiles for groups on the ideological spectrum (for example, comparing the demographics of the ideological center on each scale)?
4. Is our understanding of how ideological groups feel about major political parties and important societal issues influenced by which response scale we use to measure ideology?
Below, you’ll find the results of our experiment. For ease of discussion, I’ve labeled the interior points of the scales even though the question itself only assigned labels to the endpoints (that is, Extreme Left and Extreme Right). The resulting scale (in ascending “numeric order”) is as follows: Extreme Left, Left, Center-Left, Center, Center-Right, Right and Extreme Right.
Distributions across the ideological spectrum
The concern that fewer people would be willing to self-report as Extreme Left when that response is associated with 0 rather than 1 is unfounded. In all three countries, respondents were equally likely to pick the Extreme Left answer category regardless of which numeric scale was used. Similarly, in each country, item nonresponse rates (the rates at which people chose not to answer the ideology questions) were consistent between the two scales.
Furthermore, while Left self-reports differed across the scales in all three countries, it was not in the direction that would support the hypothesis that people would be unwilling to choose the Extreme Left category when it’s assigned a value of 0. If this hypothesis were true, we’d expect a spike of 0–6 scale respondents choosing Left (numeric value 1) as compared to the 1–7 scale respondents choosing Left (numeric value 2). In fact, more respondents who were assigned the 1–7 scale chose Left in each country.
Across all six scale distributions we studied, there was a peak near the middle that tapered toward the scale’s ends. In the UK, both scale versions peaked at the ideological Center, regardless of the numeric value associated with it. But in France and Germany, the modal category was associated with the numeric value 3, regardless of the ideological label: The number 3 represents Center on the 0–6 scale, but Center-Left on the 1–7 scale. There were also significant differences between the scales at the Center positions in France and Germany, with the 0–6 scales having more Center responses.
These outcomes suggest that when a scale is easily divided in half — for example, when the maximum value is 6 rather than 7 — it’s more likely for respondents to select the midpoint. Previous research has found that respondents are likely to assume that half the top endpoint is a scale’s midpoint, so when half the top endpoint is not an answer option (e.g., 3.5 on the 1–7 scale), respondents seeking the central point on the scale may sometime choose 3 (not the scale midpoint) and sometimes choose 4 (the actual scale midpoint). Those who received the 0–6 scale could more easily find the midpoint (3) by halving the top endpoint.
See here for supplementary tables and graphics.
Left-right lean of the scales
To determine if the differently numbered scales affected the broader left-right lean of the scale, I condensed each seven-point scale into three general categories: Left-c, Center and Right-c. (The “-c” differentiates these categories from the similarly named categories in the seven-point scales.) Left-c is a combination of the Extreme Left, Left and Center-Left categories, Right-c is a combination of the corresponding categories at the other end of the scales, and Center is simply the Center category in both scales.
In all three countries, using the 1–7 scale increased self-identification as Left-c in comparison to the 0–6 scale. Scholars have noted that the meaning of the center in political ideology scales can vary based on context and respondent interpretation. The use of 0 vs. 1 as the lower endpoint could have altered the respondent perception of the question and its answer options, leading to different distributions.
See here for supplementary tables and graphics.
Demographic differences
In all three countries, when using the condensed distribution of Left-c, Center and Right-c, there are no statistically significant differences between the two scales when it comes to demographic structure (age, gender and education). For example, within each country, the Left-c is similarly divided between old and young, male and female, and more and less educated — no matter which scale is used.
When looking at the full 7-point ideology scales, only four categories have sufficient sample sizes (roughly 100 cases or more) to make comparisons between the two scales. These are the Left, Center-Left, Center and Center-Right categories. (Center is the same as in the condensed distribution so is not considered again). Looking at the demographic makeup of these categories, there is only one statistically significant difference: Using France’s 0–6 scale, a majority of the Left is male (56%), while on the 1–7 scale, a minority of the Left is male (41%).
See here for supplementary tables.
Political party favorability and other societal issues
We also asked respondents in each country if they had a very favorable, somewhat favorable, somewhat unfavorable or very unfavorable opinion of several national political parties. For this analysis, I selected two traditional parties from each country (one party each from the left and the right): France’s Socialist Party and the Republicans; Germany’s Social Democratic Party (SPD) and Christian Democratic Union (CDU); and the UK’s Labour Party and Conservative Party. I compared “net” favorability (combining very favorable and somewhat favorable) toward the parties for each ideological category between the scales.
Overall, political ideology corresponds consistently to political party favorability, no matter which scale is used. There are a few differences across the three countries, but nothing suggests a systematic difference between the two ideology scales.
Meanwhile, to get an idea of how the two scales interact with other topics, I compared the different measures of ideology across a variety of substantive questions. The items selected were about religion (the share who say religion is very important in their life), nationalism (the share who completely/mostly agree that “our people are not perfect, but our culture is superior to others”), minority religions (the share who say Islam is fundamentally incompatible with their national culture and values), immigration (the share who think the number of immigrants to their country should be reduced) and social issues (the share who think abortion should be legal in all or most cases). The survey did not ask questions about economic policy, which could also relate to an individual’s political ideology.
By and large, respondents in the corresponding ideology categories from the different scales give the same answers to these questions. This is true for both the consolidated left-center-right categorization and the categories from the seven-point scales for which we have sufficient sample sizes to do an analysis (roughly 100 cases or more). Across all these points of comparison, nothing suggests systematic differences between the ideology scales. And the individual differences — even when fairly wide — tend not to change the overarching story of a given group’s stance on a particular topic. For example, using France’s 0–6 scale, 59% of the Left thinks abortion should be legal, while on France’s 1–7 scale, 87% of the Left supports legalized abortion. This is a statistically significant difference, to be sure, but it still shows broad support among the Left.
See here for supplementary tables.
Conclusions
If a researcher’s goal is to get a sense of the relative size of the political left, center or right within a country, there can be meaningful differences depending on whether one uses the 0–6 scale or the 1–7 scale — especially when it comes to the end of the scale closest to 0. Further research is required to know which scale presents a more valid measure of political ideology.
However, an ideological group’s demographic profile, favorability toward specific political parties and attitudes on other important issues do not drastically change when different numeric values are used in a seven-point scale. The overall impact of using a 0–6 scale or a 1–7 scale as an independent variable or control for social research appears minimal.
Jonathan Evans is a research analyst focusing on religion at Pew Research Center.
