In 2019 in the days when conferences happened in real life, at the Quirk’s London I listened to Pete Cape from Dynata talking about how the quality of research data from survey panels might be compromised. It was a fascinating 30 minutes, and certainly gave me lots to think about.
He divided the problem of poor data into three categories –
- Problems with fraud
- Problems with individual respondents
- Problems with the sample overall
For those of you who aren’t in the research industry, here’s a quick explanation of how a survey panel works.
If you need to speak to an audience you don’t have easy access to, or if you’re looking for a very specific type of person – for instance someone who wears glasses and likes broccoli (yes, I’ve gone very random I know, but it’s just an example!) – you can enlist the help of a panel provider. Panel providers pre-recruit hundreds of thousands of people who are willing and able to take part in research surveys, in exchange for a small incentive. The panel provider will send emails to respondents asking them to take part in your survey, charging a fee per completed interview.
OK, so now let’s go on to the three main problem areas –
1 Problems with fraud
This covers people who are deliberately lying or cheating. Frankly, there isn’t much anyone can do about this, other than removing these respondents when they are found. Thankfully, most people have better things to do with their time than behaving fraudulently.
2 Problems with individual respondents
This category divides into two sub-groups –
- Panellists who are engaged with the process and want to complete the survey to the best of their abilities, but for whom the data has been compromised. This might be because of poor questionnaire design, or because the questionnaire was poorly translated or written in unfamiliar language; or it might be because the quality control checks were excessive.
- Panellists who aren’t engaged with the process. This might be because they aren’t paying attention or they’re bored, or it might be because they’re only taking part in the survey for the reward.
3 Problems with the sample overall
I want to talk in more detail about problems with the third category. This raises questions such as –
Where has the panel come from? — If you are buying your panellists from a company that has only ever advertised for panel members on Website A, then your panel will never provide you with a nationally representative sample. It will just be a sample that’s representative of visitors to Website A.
How were the individuals recruited? — If the panel was recruited on the basis of a large joining incentive, then it’s possible that the individuals are not engaged with the research process.
How were they vetted? — If individuals were allowed to join the panel without any kind of verification or vetting, then it’s possible that some of them might not be who they seem. So they won’t actually be your target audience.
How were they selected? — If individuals are emailed time and time again, they are likely to become fatigued. Check how often individuals are allowed to take part in surveys.
How are the panels maintained? How often is the data checked? — As an example to show the importance of checking data, someone might have joined the panel two years ago when they were single and child-free, but now they have a partner, a baby and a dog.
To conclude
In essence, the people who take your survey will only ever be representative of the overall panel.
Thankfully, more and more we are seeing panels coming together from multiple sources. As a result, it’s much more likely that there will be better representation on the panel. Diversity of representation is something to be mindful of when you are choosing your sample provider.
If you need advice about how to select a sample provider, then get in touch – I’d be happy to chat through the options with you.
