POLITICS
Is it really “the economy, stupid”?
The limits of using polls to explore the relationship between economic circumstances and vote intentions
If you’ve paid any attention at all to the news recently, you have probably seen more than a few stories about how the economy is weighing down President Biden’s reelection hopes. Many of these stories are based on data from polls that ask people about their own economic situations and also ask them what they think about Biden or how they plan to vote in 2024.
Such an analysis may look something like the following graph, which plots responses to a question we asked on the Cooperative Election Survey about how each person’s household income has changed over the past year and their approval rating of President Biden.¹ From the graph, the relationship appears fairly straightforward. People whose incomes have been stagnant over the past year give Biden a 39% approval rating, but those with incomes that have decreased rate him much lower and those whose incomes have increased rate him much higher.
But we also see a potential red flag when we look at this data: more Americans reported that their incomes decreased rather than increased over the past year despite the fact that government data indicates that wages and salaries are on the rise.
In fact, 17% of Americans reported that their income decreased somewhat in the past year while another 10% said that their income decreased a lot. Only 19% of Americans reported an increase in their income. Even more striking is this: Republicans were much more likely to report that their incomes declined during the previous year compared to Democrats. 35% of Republicans reported a decline in household income compared to 19% of Democrats.
Is it really true that Republicans are struggling significantly more than Democrats when it comes to their household incomes? Or is this another example of a pattern that survey researchers call “expressive responding” — a phenomenon where individuals strategically provide dishonest answers to survey questions in an attempt to make their party look good or the other party look bad?
A Republican answering our survey might consider saying they are doing worse economically than is actually true as a way of supporting the narrative that the economy is struggling under the Biden presidency. Likewise, Democrats may report that they are doing better economically than they are to undermine that same narrative.
It is often hard to detect survey respondents who are engaging in such expressive responding because we don’t actually know when someone is giving dishonest survey responses. (Though see here for work I did with Sam Luks where it was pretty clear). But in the case of income change on the 2022 CES, we actually do have data that allows us to get a good sense of this because it just so happens that 11,000 of our respondents were individuals we had previously interviewed back in 2020. Each time we interview a respondent, we ask them to report what their household income actually is. So for each respondent we know what they said their household income was in 2020 and what they said it was in 2022.² Since these questions provide precise response categories and are buried among other demographic questions, it is unlikely that respondents would think to be dishonest when answering them in the same way that they might for the more vague and politically relevant question about income change.
So what do we see when we explore what people reported their income was in 2022 compared to what they reported it was in 2020?
Here the picture looks quite different: only 18% of Americans gave a lower household income in 2022 than they did in 2020, while 35% reported a higher income level. Additionally, the partisan differences on this metric are much smaller — 21% of Republicans reported lower incomes in 2022 compared to 2020 while 18% of Democrats reported the same. Of course, it is important to point out that these two questions are not completely comparable since the self-reported change question asks about the past year but this second approach compares incomes reported over a two-year period. Nevertheless, it is doubtful that too many people will have suffered a decrease in income from 2021 to 2022 if their income had increased from 2020 to 2022.
I can also use this data on each respondent’s actual self-reported income and compare it with how they claim their income has changed. I do this separately for Democrats and Republicans in the following graph. The panel on the left shows the percent of Democrats who reported that their income decreased, increased, or stayed the same during the previous year plotted against a measure of how much their self-reported income had actually changed between 2020 and 2022.³
Democrats whose incomes had decreased since 2020 were almost as likely to say that their income had increased during the past year as they were to say that it had decreased. This would be consistent with an expressive responding pattern, where some Democratic respondents who had seen a drop in income may have been unwilling to admit that for fear of reinforcing narratives about a poor Biden economy. Note, however, that Democrats who had higher incomes in 2022 than they did in 2020 were much more likely to report that their incomes had increased rather than decreased.
With Republicans we see an opposite pattern and one that is even more pronounced. Republicans who reported lower incomes in 2022 than in 2020 were much more likely to say that their incomes had decreased during the past year. But what is striking is that even Republicans who reported significantly higher incomes in 2022 compared to 2020 were still more likely to say that their income had decreased during the previous year than to say that it had increased. Again, this disconnect would fit with the idea that some respondents are engaging in expressive responding by reporting that their incomes were lower even when that is not true simply to reinforce the narrative that the economy under Biden is struggling.
It is important to note that the responses plotted in these graphs are not wholly divorced from reality. Republicans whose self-reported incomes had increased between 2020 and 2022 were less likely than other Republicans to report that their income had decreased during the prior year. But the issue is that many respondents are not providing honest responses, and they are doing so in a way that appears to be driven by their partisan loyalties. The unfortunate effect of this pattern is to exacerbate the kind of relationship we saw in the first graph, making it look like income change is having a major effect on Biden’s approval rating when in fact it is just as likely that how somebody feels about Biden is affecting how they answer the question about income change.
I can show this most clearly by recreating the first plot, but this time using the measure of how each respondent’s self-reported income actually changed between 2020 and 2022 rather than how they said it had changed during the previous year. Using this approach, it turns out that the relationship between income change and Biden approval almost entirely disappears. People who report household incomes that are at least $40,000 greater than what they reported in 2020 are only marginally more approving of Biden than those whose incomes actually decreased during the same period. From this graph, it appears there is not much there when it comes to the relationship between how people’s incomes actually changed during the first two years of Biden’s presidency and how much they support the president.
This is not to say that the economic picture is completely irrelevant to Biden’s relatively low approval rating or his reelection chances next year. It is reasonable to suspect that some swing voters are being persuaded by high inflation. But what this analysis shows is that simply asking people how inflation is affecting them and then comparing that to how they might vote in 2024 is not a good way to establish an accurate picture of that relationship. Ultimately, we cannot always take at face value what people tell us in polls to support a narrative that is circulating in the news. After all, many Americans are aware of these narratives and are often all too eager to bend the truth when answering polls with the aim of making their own party look better and the other side worse.
¹ The question specifically reads: “OVER THE PAST YEAR, has your household’s annual income…? Increased a lot, increased somewhat, stayed about the same, decreased somewhat, or decreased a lot.”
² The question specifically reads: “Thinking back over the last year, what was your family’s annual income?” There are 16 income categories to choose from ranging from “Less than $10,000” to “$500,000 or more”. Some respondents decline to give their income when answering this question, so they are excluded from this analysis.
³ To calculate actual change in self-reported income I subtracted the median value of the income category selected in 2020 from the median value of the income category selected in 2022.
