Who really fills in a survey?
The challenges of gathering quality data at scale via panel providers
What are panel providers?
Panel providers are organisations that have gathered together a number of individuals who are open to participating (usually via survey) in a variety of research projects. They offer a convenient solution to organisations by:
- Finding and accessing the appropriate population for your project
- Distributing your survey to them
- Guaranteeing a specified number of responses
This is particularly useful for organisations who wish to conduct market research outside of their existing audience. For example, you may wish to develop a product for a specific audience, but you may not readily have access to that population. It would be a very time-consuming challenge to find and engage people in this space and that’s where panel providers offer to do the heavy lifting for you.
However…
However, panel providers are rarely held accountable for the quality of their data. They may cut corners to remain competitive, resulting in questionable data quality.
One of our experiences illustrates the scale of the problem. We enlisted the help of a panel provider for one of our projects. They claimed to provide very thorough quality assurance and vetting processes to their panel. However, given the investment, we insisted upon including some ‘trap’ questions to weed out disingenuous respondents. This basically involved including some fictional publications within a closed list of real publications and screening out anyone who selected one of these.
This simple technique weeded out 60% of all respondents who started the survey!
It’s very hard to maintain confidence in the data when the majority of respondents are getting flagged as disingenuous.
So why work with panel providers at all?
The above statistic is scary and it is certainly true that not all panel providers are created equally. Some will carry out more quality assurance checks than others. Some will charge very little and some will charge a lot. Some will tell you that they can access ‘any’ research population, others will be more honest and admit they may struggle to find the thousand US C-suite executives you need to fill out your survey.
However, finding and accessing the specific research population for your project is an immensely difficult task, especially against a background of increasing research fatigue and the number of communications vying for an individual’s attention. It still makes sense to outsource aspects of this work, but for anyone considering engaging the services of panel providers you need to go in with a) a clear idea of what you’re trying to accomplish and b) a healthy degree of scepticism, and knowledge that empowers you to hold them to account.
Below is my guide to anyone considering working with panel providers.
Tips for working with panel providers:
- Start with Qualitative research
Some people turn up their nose at qualitative research and accuse it of being too granular and not ‘statistically significant’. This is to miss the entire point of qualitative research, and in regards to market research, it is an absolutely vital starting point. It gives you a brilliant base for a) identifying emerging themes, b) getting a sense of how important something is (or isn’t) and c) getting a feel for how difficult it is to engage the specific research population you wish to solicit feedback from at scale e.g. if it takes 3 months to get a focus group together of 10 C-suite executives then imagine how difficult it would be to get 500 of them to fill out a survey…
2. Think of how much feedback you realistically need
People talk a lot about statistical significance and in my opinion this often obfuscates the real challenge which is gathering quality data, and in particular “are these the right people to fill out my survey and can I trust they’ve completed it in good faith?”.
If the data quality is poor then statistical significance is not going to make it any more reliable, in fact, it risks giving people misplaced confidence in misleading data. I would argue that this is worse than no data at all and much worse than a well-conducted piece of qualitative research.
Take a step back, and ask simply, how much data do I need to be confident in the decision I need to make? As a general rule of thumb with market research, I subscribe to the “Quality 100” rule. If you can get 100 quality responses from your target population then that should give you a reasonable amount of confidence in which to make your decision.
Now, this is by no means perfect, and I’ll be clear that this needs to be applied to each segment you wish to analyse your data by. It’s also evidently not appropriate for certain types of research (e.g. medical) but I will say from experience, where most market research is as much an art as a science, the “Quality 100” rule will serve you fairly well.
3. Be prepared to ask how?
Panel providers are usually quite good at sourcing ‘gen pop’ respondents. Often, they utilise financial incentives to encourage people to take surveys e.g. fill out 5 surveys and earn a £2.50 retail voucher. Think about the respondents who this would attract, and how many would be motivated to complete your survey in good faith, rather than clicking through as quickly as they can?
It is always worth asking how because if the panel provider can’t answer this satisfactorily, it suggests that they will not be that bothered about quality assuring their own data.
A good example from my experience is a project we carried out where we were looking to recruit Black participants to take part in a specific project. They had to work in similar industries to our target market and also had to hold a minimum of an undergraduate degree. All in all, a relatively specific demographic.
When we ran an RFP to outsource this, almost every provider said that they could recruit these participants, but ultimately only a couple gave a satisfactory answer to the question ‘how?’. We ended up working with a specialised Black-owned provider who’d built up a great network over a number of years. Other, often bigger, providers had given an answer more along the lines of “we’ll find a way”. In the end, we went for a more qualitative approach as a starting point but gathered some key findings that put us in good stead should we need to validate these further down the line. Most importantly, we have complete faith in the data we collected and a benchmark from which to compare future work.
4. Be prepared to pay
The more niche the population, then the harder it will be to recruit. In these cases the incentive or level of effort required is more and this usually comes with a cost. Lower-cost panel options can be helpful if you’re looking for a fairly generic respondent base but for anything a bit more targeted, it will usually require a specific strategy to engage various networks and the price will reflect that. Please note though, that a high-cost sample is no guarantee of quality. I’d still highly recommend ensuring you have a clear quality assurance and review process in place for any engagement and ensure this is agreed up front.
Hopefully the above tips help anyone seeking to engage with panel (or sample) providers in the future. Please share your thoughts in the comment section below!
