Polling Both Sides of an Issue: Different Questions, Different Results

Last month, Change Research conducted a poll of 1,182 registered voters in Idaho on Idaho Senate Bill 1159, a proposed law that would increase the requirements for qualifying an initiative or referendum on the ballot, thus making it significantly more difficult for Idahoans to make their voice heard in how the state is run. Our poll was performed on behalf of our client, Idahoans for Free and Fair Elections.

Our poll found that:

• 69% of voters opposed SB 1159, and only 17% of voters supported the bill after reading the following description:

“The Idaho Senate just passed SB 1159 by 1 vote. The proposed law makes it far more difficult to qualify for a statewide ballot initiative in Idaho. The bill shrinks the signature gathering period from 18 months to six months, increases the number of signatures from 6% of registered voters to 10%, requires signatures equal to 10% (up from 6%) in each of 32 of the 35 legislative districts from the current requirement of 18 of the 35 legislative districts, and other measures to generally make it more difficult for Idahoans to put ballot initiatives on the ballot. Do you support or oppose Senate Bill 1159 becoming law in Idaho?”

• 89% of voters believe the initiative process is an important constitutional right that should be protected.

Our poll received news coverage comparing it to another poll performed on behalf of the Idaho Farm Bureau Federation, whose findings sharply contrasted ours. Specifically, the other poll found that:

• 83% of voters agreed with the following statement:

“Rural communities should have an equal voice to larger cities and counties in the state in determining which propositions qualify for the ballot.”

Essentially, our poll found that 69% of voters opposed SB 1159, and the opposing poll found that 83% of voters agreed with the underlying purpose for passing SB 1159.

We found this situation to be an interesting methodological case study, and decided to run another poll using a replica of the Farm Bureau’s survey instrument. We sought to determine whether the contrasting results were due to their phone polling method versus our online method, or due to a difference in question phrasing. Our suspicion was the latter.

In our follow-up poll of 748 registered voters (conducted between April 12–13, 2019), we found that:

• 79% of voters agreed with the statement on “rural communities” used in the Farm Bureau poll, which is not meaningfully different than the Farm Bureau’s result.
• 58% of voters opposed the specific language of SB 1159, and just 14% supported it after reading the same description of the bill as was in our original poll. Once again, the ratio of oppose:support, about 4:1, matched our original result, meaning that reading the statement about rural communities did not meaningfully affect voters’ opinion on SB 1159.

This test confirms that voters can simultaneously believe that rural communities should have equal voice to larger cities, and that SB 1159 is a bad idea. The two statements are not directly related.

More broadly, this test confirms our methodological point: when polls ask about an issue in different ways, it is important to take those language differences into account when synthesizing those polls and acknowledge that they can not be directly compared. In this case, the Farm Bureau poll asked a sentiment-based question on a larger philosophical issue, while our poll asked specifically about support for SB 1159. In comparing results from these two questions, one cannot assume that drastically different results invalidate either poll; they simply ask different questions.

Change Research provides fast, accurate, and affordable public opinion polling. Find a full report of our polling accuracy here.