Collecting Community Survey Data During COVID-19
by Mychal Cohen
A publication of the Local Data for Equitable Recovery Resource Hub
Collecting primary data is key to understanding the world, but survey response rates have been declining in recent years. Non-profits rely on the data that come from these surveys to better understand their constituencies. Crises intensify the need to better understand people’s thoughts, beliefs, and circumstances, but also introduce new obstacles to collecting and reporting accurate data. The pandemic has exemplified this struggle, rapidly changing many households’ income and their need for and access to resources, including health care and child care, healthy food, and safe transportation. Local organizations have faced a deficit of information that makes it difficult to respond to these needs.
Many of the organizations involved in the Using Data to Inform Local Decisions on COVID-19 Response & Recovery grant program have addressed this information deficit by collecting new data through community surveys. By adapting their data collection, organizations have overcome struggles associated with reaching people whose time and energy have been strained during the pandemic. The lessons they learned can inform best practices for data collection efforts during and after the pandemic.
Where possible, flexible sampling methods may be necessary to get responses
Most partners identified early on that their methods would need to be flexible to get the number of responses they needed to have viable data. Although more complex and rigorous sampling strategies would have increased their ability to discuss subpopulations and to eliminate potential sources of bias, the negative effects of low response rates would have been increased. Most partners used more nonprobability sampling (where a researcher doesn’t know a respondent’s share of the total population) to allow for more flexible outreach.
Partners who conducted interviews or surveys pursued snowball samples, where respondents direct researchers to people in their networks to survey, or convenience samples, where respondents are those the researchers can connect with through outreach. Responses to these sampling methods tend to have larger amounts of bias than more complex sampling techniques. For instance, a snowball survey might miss subpopulations that are less connected or more marginalized. Comprehensive outreach, diverse partners, and thoughtful targeting can mitigate some of the shortcomings of these methods.
Developing relationships is key
Grassroots relationships with people and communities and relationships with trusted organizational partners allow a researcher to avoid being perceived as an outside entity with no actual stake in a community’s conditions. Forming relationships with the communities you are studying helps you avoid developing an extractive relationship and instead helps you collect data in partnership with community members. In addition, building relationships can improve your reach and make community members more likely to make time for surveys or interviews with unknown people or organizations.
Catalyst Miami surveyed business owners in North Miami to find out how the pandemic had affected their businesses financially, whether they were able to access relief funds, and what their expectations were for the future. However, they struggled with low response rates: early efforts to conduct phone outreach yielded an initial influx of responses that plateaued a few months into data collection. Catalyst Miami attributed this in part to the time constraints that small business owners faced during the pandemic. The pressure of staying afloat meant that business owners didn’t have the time to complete a survey. Catalyst Miami’s initial outreach relied on preexisting community relationships, some limited door-to-door canvassing, and phone outreach. In order to reach their target number of responses, they needed additional strategies, and two decisions helped them reach their final goals. First, they changed the delivery format by distributing the survey via email, after realizing the survey was email friendly. Second, they collaborated with the North Miami Community Redevelopment Agency (North Miami CRA) to expand their outreach. The agency leveraged its own relationships with business owners to convince them to take the survey. Even though Catalyst Miami had their own network, expanding it through the North Miami CRA allowed them to reach new businesses and get more responses from those they had already contacted. Without preexisting community relationships and without leaning on an organizational relationship with the North Miami CRA, Catalyst Miami would not have been able to reach their response goals, and now has information they need to target technical assistance to help business owners get back on their feet.
Tap in to existing service points
Another strategy for building relationships is to connect with potential survey respondents in places they already are. As with any nonprobability sample, it is important to understand the biases that this introduces and how this affects data interpretation. In this case, focusing on existing points of connection provides a specific subset of the people in need: those who know about existing services and are able and willing to access them. Depending on the situation, that population may be a large or narrow share of those in need. This should be directly addressed in any reporting of findings.
Para Los Niños surveyed residents receiving services in Los Angeles to better understand the impacts of the pandemic, particularly on access to food, transportation, and health care. To administer the survey, they worked with 13 partner organizations that were already providing services and so had a natural connection point with residents. By distributing the survey in person at service points, Para Los Niños and their partners avoided technological barriers introduced when surveys are only distributed online and via email. Para Los Niños understood the bias that such a sampling strategy would introduce into their analysis: households that may have been in need but were not connected to services would not show up in the survey results. Though it would be impossible to fully eliminate this limitation, Para Los Niños was intentional about expanding the reach of their partnerships, reaching residents they had not had prior contact with, and doing proactive outreach. This model of directly connecting with residents proved very effective for outreach, particularly because their goal was to solicit the experiences of residents who were receiving services. The information Para Los Niños collected will be shared in an upcoming town hall, where they will lead a community visioning process to create a community bill of rights.
Concluding Thoughts
Collecting primary data fills critical gaps in knowledge about the needs and perspectives of people suffering most from the effects of the pandemic. Collecting such data during the pandemic presents challenges, but flexibility in methods, outreach, and analysis and an understanding of tradeoffs between accuracy and bias are key to researching during times of crisis.
We thank Catalyst Miami and Para Los Niños for their contributions to this blog post. The two organizations are grantees of the Robert Wood Johnson Foundation’s Using Data to Inform Local Decisions on COVID-19 Response & Recovery program.
