(Un)Accountable Poll Participants

Why do the polls differ on Trump’s job approval ratings?

Over on FiveThirtyEight, Nate Silver pointed out last week that Trump’s average approval rating for the month of February follows a peculiar pattern: Trump’s approval ratings are higher in polls of voters (as opposed to all adults) and in polls using automated or online survey methods.

Trump’s approval ratings are systematically higher in polls of voters — either registered voters or likely voters — than they are in polls of all adults. And they’re systematically higher in polls conducted online or by automated script than they are in polls conducted by live-telephone interviewers. — Why Do Polls Differ On Trump’s Popularity

Silver’s take is that the gap between all adults and voters will likely diminish with more data. He remains skeptical of the “shy” Trump supporter theory as a potential explanation for the differences between live caller and online or automated survey methods. He also questions the reliability of automated polling methodologies and whether there is enough data for a definitive explanation. Nevertheless, if these differences are pervasive, even if they narrow in the future, what could explain this gap?

As of today¹, and since Silver’s originally post, twelve additional surveys reported their results and the gap has either widened or remained intact.

• The Survey Population averages for approve now stand at 42% for all adults and 50% for voters, widening the difference by 3 percentage points. Results for disapprove are 52% for all adults and 48% for voters, widening the gap by 1 percentage point.
• For Survey Method, the approve averages are 42% for live caller and 48% for online or automated; and for disapprove, averages are 51% for live caller and 48% for online or automated. Both gaps only changed by 1 percentage point or less.

Accountability in Survey Populations

All Adults vs. Voters (registered or likely)
One possibility is that voters perceive the question of whether they approve or disapprove of President Trump’s job performance as a postdecisional accountability exercise. Philip Tetlock’s research² finds that after-the-fact demands for accountability are likely to motivate you to justify a previous decision, maintain that you made the right choice, and interpret new facts as supporting your original position. And while there are issues with the demographics of the voter survey population, it is unlikely in the near-term that this group will be able to separate their vote from how they evaluate the president on his job. Meanwhile, the non-voters included in all adults, are more likely to wrestle and incorporate negative media reports and new information into their judgement of how well the president is doing his work.

Accountability Contexts of Survey Methods

Live Caller vs. Online or Automated
A glaring limitation in trying to isolate what is causing the differences in the survey method results is the potential “cross-contamination” with the survey population.

For whatever reason, most live-caller approval ratings polls survey all adults, while most automated or online polls survey registered or likely voters. — Nate Silver

However, I think we can proceed as the effects related to the timing of accountability demands can be understood alongside the effects of the context for those demands. Previous research supports that you are more likely to engage in open-minded critical thinking when, before forming any opinions, you expect to be accountable to an audience “a) whose views are unknown, b) who is interested in accuracy, c) who is interested in processes rather than specific outcomes, d) who is reasonably well-informed, and e) who has a legitimate reason for inquiring into the reasons behind participants’ judgments” (Lerner & Tetlock 1999). While I recommend caution when generalizing these findings, you will probably agree with me that much of the criteria overlaps with the experience of answering a live telephone survey. Typically, a survey participant will receive a questionnaire from a pollster a) whose views are unknown, b) who is interested in accuracy, c) who is interested in knowing your opinion not advocacy, d) who is reasonably well-informed, and e) who has a legitimate reason for inquiry.

All survey methods assure participants that responses are anonymous, but the accountability context of answering live to a real person may motivate more thoughtful answers. While plenty of work has been done on how to leverage technology as a control for accountability systems, little has been done on the effects of technology-mediated accountability contexts. With no active measures in place for motivating thoughtful answers, the automated and online surveys may be no better than the anonymous nonaccountable condition on many past studies. Some results indicated that participants spend less effort in considering their answers, and perhaps basing them on impulse and bias³.

Notes

1. Based on polls conducted between Feb. 1 and Feb. 27, 2017 which include an additional 12 polls since Silver’s post. Source: Huffington Post Pollster: Trump Job Approval
2. “There are many reasons to suppose, then, that postdecisional accountability — far from encouraging complex, self-critical thought — actually exacerbates many judgmental biases and defects of the cognitive miser. Demands for accountability sometimes motivate people to bolster previous decisions, to be overconfident in the correctness of those decisions, to overassimilate new evidence, and to deny difficult value trade-offs, particularly when the trade-offs require acknowledging flaws in one’s past decisions” — Tetlock, P. E. (1992). The Impact of Accountability on Judgement and Choice
3. “Accountability has been shown to inhibit racially biased decisions, as illustrated in an interesting study by Ford, Gambino, Lee, Mayo, and Ferguson (2004). In this field experiment, White sales managers were given the résumés of applicants who were told to be either African-American or Caucasian. Those sales managers who were not in the accountable condition were more likely to recommend hiring the White applicant although the résumés were similar. However, the sales managers in the accountable condition were more likely to rate Black applicants more positively.” — Hall, A. T., Frink, D. D., & Buckley, M. R. (2017). An accountability account: A review and synthesis of the theoretical and empirical research on felt accountability