Research About Research: Key Takeaways from AAPOR 2023

by Sheldonn Plummer-Duff and Rosemary Cosgrave

Strategic insights drive Bully Pulpit Interactive’s approach to designing winning campaigns. We use research to uncover how to frame issues, win arguments, and make change for our clients. A core group of our insights and survey operations team brought our expertise on the road to Philadelphia, and presented two panels at The American Association for Public Opinion Research (AAPOR) Annual Conference.

Here’s some of what we learned:

Maybe Gen Z does answer their phone.

In analyzing 4.2 million contact attempts, BPI’s insights revealed how different groups are responding to survey modes — giving us a framework to better structure a sample so we reach people where they’re most likely to respond. Our findings showed us that age does play a role in responsiveness, but not in the ways we might expect.

Young respondents had higher completion rates when called on cell phones — and were less likely to complete on text-to-web (TTW) surveys than other age groups. On the other hand, older respondents completed TTW at higher rates.

Beyond our own research, the rest of the conference showed us that TTW surveys continue to be a valuable addition to any data collection, but shifting to TTW isn’t a panacea that addresses all the issues of representativeness. Our findings and these sessions continue to prove that BPI’s multi-mode approach and structuring a sample that maximizes who we can get by mode, is the best way to ensure we reach as wide a pool as possible.

We still need to earn people’s attention.

While panelists agree to take surveys on a regular basis, we cannot take their attention for granted. Hearing directly from respondents during Dynata’s Voice of the Respondent session underscored the importance of creating surveys with the respondent’s experience in mind. Hearing respondents’ feelings on how the design of a survey can make them feel, at what point we lose their attention, and the tactics they take when bored highlighted how inattentiveness can present itself in ways that aren’t caught by standard attention checks.

This session showed us that respondent attentiveness is more important than ever — which is why the BPI team works to discover innovative ways to improve a respondent’s experience with our surveys: keeping screening sections and surveys succinct, building in mental breaks, considering the burden of certain sections and questions may pose, and more. We see the impact of these optimizations every day with reduced dropout rates (~10%), better data quality, and clearer data results and it was great to hear from respondents themselves that they value this.

Success is weaving DEI into the core of your organization and research.

As part of her role on AAPOR’s Inclusion and Equity committee, Sheldonn participated in a roundtable discussion on DEI initiatives at research organizations. The panelists discussed pushing for inclusion, both with and without a formal DEI title, as well as how to measure the success of DEI initiatives. Going beyond how organizations can implement DEI, the discussion focused on the underlying tactics that make DEI efforts truly successful: pushing for retention, not just recruitment; approaching research with a diverse team; and mitigating biases to reach a representative group of respondents.

A recurring sentiment across all panelists was that DEI is here to stay and this is how leaders can weave it not only into the core of their organizations but into their research methodologies to avoid getting left behind. The roundtable also presented a preview of AAPOR’s draft IEC report that Sheldonn is co-authoring. The final report will measure AAPOR’s DEI initiative and members’ priorities for DEI within the association and the wider industry.

Representativeness was a key theme of both BPI’s contribution to AAPOR and the sessions that stood out to us. From expanding on our use of mixed-mode recruitment methods to better reach less represented populations, continuing to take a critical look at our questionnaire design to make sure it’s accessible to all respondents, and incorporating a DEI lens to reduce bias, we left AAPOR with a feeling the industry is asking the right questions but there’s more to do and we cannot wait to dive in!