OPINION18 May 2015

Throw the panel in the river


The problem of survey respondents has been solved, or so many in the market research world would have you believe.


They’ll tell you that online respondents, whether as part of a retained panel or an ongoing stream or ‘river’ of recruits, are plentiful, cheap and willing to complete endless surveys for money or fun. The challenge, therefore, is now more about research methodology than recruitment.

This view is superficially attractive but, as someone who works with online respondents every day, I’d argue it’s flawed. While it might suit the needs of some to view respondents as commoditised, they are, in fact, human beings who get bored, who cut corners and who need motivating – and they need to be treated as such. If you want robots to complete your surveys then I can probably create you an algorithm that will tell you what you want to hear!

This misunderstanding of the nature of respondents isn’t just an issue for companies like mine that are in the business of supplying audiences for online surveys. It matters to the wider market research industry who rely on these audiences and it matters to the brands who use our research to make product and marketing decisions.

Panellists are not a commodity

The issue of online respondents actually stems from their perception as a commodity. Researchers believe that panels will answer any questions they are given accurately, efficiently and with care. With that thinking they come up with ever longer, ever more complex questionnaires for respondents in order to create more monetisable insight.

However, the length of a survey has a direct link to the accuracy of the results. Almost 60% of Instantly panels spend more than 15 minutes answering surveys at a time, and almost 80% would consider a survey longer than 15 minutes. However, many admit to occasionally speeding through the process, citing boredom or lack of/low incentives. Almost a third of respondents have said they’ll click random answers if they are presented with too many options.

With survey times now regularly exceeding 20 minutes (when 15 minutes seems to be an upper limit for many respondents),  incentives need to be increased in order to capture high quality data and avoid this trend of speeding through surveys. It is clear that research outcomes are being impacted by a failure to pay attention to respondents in the methodology. Of course, you can pay more to incentivise people to do longer surveys but, by doing so, you end up focusing solely on an audience of people who are motivated by the reward, again impacting on the results.

So, what are the solutions to this issue? How can researchers, brands and panel companies ensure that they’re getting the best out of panels and therefore the most accurate insight into their opinions?

Not all panels are equal

The first thing is to recognise that not all panels are equal. For a long time competition for research audiences has been on price, but that’s not going to optimise your research outcomes. If you look at the audience in detail you can understand where the value really lies. What is the recruitment methodology? Can respondents be tracked back to source? What are the dropout rates for the panels? All of these indicators will show the value of a panel in terms of quality of audience.

Secondly, short is good. Time is precious, even for people answering surveys online, and the quicker you make the process the more likely the respondents are to remain engaged throughout your questionnaire. Rather than piling the questions on to create greater predictive value, work out what you need to know and what you already know and ask only the pure minimum, because this is what’s going to get the best results.

Finally, think mobile at all times. Too many surveys aren’t mobile optimised and – in today’s mobile-first world – that’s just not good enough. Last year, Ericsson predicted that by 2020, 90% of the world’s population will have a mobile phone. Respondents don’t want to sit at a desktop for 30 minutes answering questions – cut it to 10 minutes and send it to a mobile device and the results will be richer and more accurate.

Online panels aren’t broken – they still offer the most cost-effective, fast and accurate route to creating research insight. However, until research methodology recognises that respondents aren’t robots, the industry won’t be extracting the maximum value from them.

Ben Leet is UK MD at Instantly


8 years ago

Thanks Ben, good reminder on panels. One of the things you've missed in your 'to-do' list, is do design engaging and interesting surveys. And yes, agree they need to be short. But a badly designed survey even if short will ultimately disengage your respondents and get you 'bad' results.

Like Report

8 years ago

Thanks for the reminder that quality data depends on quality surveys. We all think the surveys we are write are perfect but no one is perfect. I would love to know who thinks or says that online respondents are willing to complete endless surveys for money or fun. If it's a panel owner, then they are really out of touch with the industry and are misleading their clients. I'll have to disagree with the statement that online panels offer the most accurate route to creating research insight. There are many research methods out there, both qual and quant, all of them meeting very specific needs, none of them more accurate than the other when used properly and for the right purpose. For those who aren't sure what is important when choosing a panel/sample provider, be sure to check out ESOMAR's panel guidelines. It's perfect for newbies who aren't sure what to ask about and for experienced researchers who need a quick reminder. https://www.esomar.org/uploads/public/knowledge-and-standards/documents/ESOMAR-28-Questions-to-Help-Buyers-of-Online-Samples-September-2012.pdf

Like Report