A price worth paying?

Persuading people to take part in online surveys is a major industry challenge. Does the price have to be right? Or should we avoid paying at all costs? Nick Sparrow of ICM and Angus Reid of Vision Critical argue it out

From: Nick Sparrow
To: Angus Reid 
Subject: Online Respondents
When we first met a few years ago, I’d guess neither of us foresaw the explosion in online research, or the problems it would bring.

The principal difficulty springs from the fact that interviewers in the pay of research agencies have been replaced by online panellists. They do a job similar to, say, clerical workers – answering survey questions for low pay and/or prizes they stand little chance of winning. They respond from home (or place of work) completely unsupervised and uncontrolled, answering questions and completing tasks they will sometimes find difficult to comprehend.

With little if any evidence of quality control, the logical way for a panellist to maximise earnings is to join several panels, work fast and play the system, paying little attention to the questions asked or indeed to the answers they give.

Research agencies love online research because it is very profitable, clients because it is cheaper than traditional research. But a growing number of researchers have, in my opinion, valid concerns over the reliability of the research data produced. We all have a longer-term interest in ensuring panellists provide carefully considered views, and whichever way you look at it, the quid pro quo is fair pay too.

From: Angus Reid 
To: Nick Sparrow
Subject: Re: Online Respondents
I disagree with you that online research has brought problems. As a device for collecting information from survey respondents it represents a colossal leap forward on several fronts. First, it allows for pictures (moving and still), animations, fly-throughs, contingent question wording and ordering and a host of other features that we are only now beginning to appreciate. Second, from the perspective of respondents, it offers the promise of a far more convivial way of doing surveys compared to telephone or face-to-face methods. And finally, online research is the first technique to provide a relatively easy tool for doing longitudinal research – a technique whose power most MR professionals are just starting to grasp.

It is still early days in the online research era, and the tent of offerings is very big and chaotic. There are lots of charlatans and snake oil salesmen prepared to sell online respondent by the gross without regard for any quality considerations. And there are lots of 14-year-olds posing as cardiologists to cash in on big incentives. But that doesn’t mean we should declare the online experiment as a failure and move backwards to the dreary world of telephone surveys that (at least in the US) involve refusal rates approaching 80%.

From: Nick Sparrow
To: Angus Reid 
Subject: Re: Re: Online Respondents
Efrain Ribeiro of Ipsos recently explained the need for, and importance of, the Advertising Research Foundation (ARF) online quality initiative in the US, aimed at providing answers to some current issues. The genesis of the current predicament, he explained, is to be found in explosive growth in online research. Among other things he suggested that low barriers to entry had encouraged non-researchers to start trading, and meanwhile best practices have not been standardised. He also mentioned the problem of ‘heavy responders’ in addition to a chorus of advertisers echoing quality concerns and low public confidence in online research.

I entirely agree that there are projects for which online research is ideal, and rejecting online would indeed be a step backwards. But we nevertheless need to address real concerns about online data quality and in my view that starts by building a contract of trust between researchers and their new part-time employees, the panellists themselves. The contract: fair pay for considered answers.

From: Angus Reid 
To: Nick Sparrow
Subject: Re:Re: Re: Online Respondents
There are good and bad panels. I think it’s unfair to characterise them all the same way. A recent survey of our Canadian panel revealed that almost half would continue taking surveys even if they weren’t paid. That’s because at least half of our surveys deal with social, economic and political issues and the commercial content is carefully controlled. Also an increasing number of clients now run their own panels of customers. These clients have a huge incentive to do short, meaningful surveys so as not to annoy their customers. And the customers often have an intrinsic motivation to participate – quite independent of pay.

Online panels are to our industry what moving pictures were to the entertainment industry almost 100 years ago. Remember there were a lot of really bad movies before they started to get it right. There still are many bad movies, of course, but almost all are better than what came before.

From: Nick Sparrow
To: Angus Reid 
Subject: Re: Re: Re: Re: Online Respondents
When asked to choose between a more interesting survey that offers no reward and a less interesting survey with a £2 reward, 80% of an online panel opted for the money. Heavy survey-takers, interviewed face-to-face, told us they are not motivated by the money, but several had detailed accounts of what panels they were signed up to and how much they had earned. Panellists may say they are not motivated by money – but they don’t mean it.

For sure, some panellists who are passionate about politics and social issues will want to participate in opinion polls, whether or not they get paid for their views. The same applies to surveys about books, or motor cars, or other products and services. Those who are passionate, engaged and activated need no incentive. But it should be clear that the results are unlikely to be representative of all voters or customers.

Online research using panels could, as you suggest, be compared to the early movies, some good and some very bad indeed. The problem for clients using research to make important business decisions is that, unlike with a movie, you can’t always tell just by looking whether the data is any good or not.

From: Angus Reid 
To: Nick Sparrow
Subject: Re: Re: Re: Re: Re: Online Respondents
There are huge quality differences in online panels around the world. But not all are populated by professional respondents. At Vision Critical we support about 250 custom panels for airlines, sports leagues and magazines. Many of these panels have very high levels of participation and very low rewards.

The problem facing the entire market research business, whether online or offline, is how best to secure the cooperation of potential respondents in a world where more and more people don’t want to donate their time to strangers. The building blocks of cooperation and trust are more than incentives; they are courtesy, empathy and clarity.

Some panel companies treat respondents as robots, subjecting them to intolerable surveys in exchange for a little cash. No surprise that they get professional respondents. Others are engaging a broader cross section of the market with fewer, better surveys, and enlightened communications. My experience as a buyer is that they deliver a better product.

How are clients supposed to navigate the wide fluctuations in quality in an industry where everyone anoints themselves as ‘professionals’? Very carefully. 

From: Nick Sparrow
To: Angus Reid 
Subject: Re: Re: Re: Re: Re: Re: Online Respondents
People join custom panels because of their interest in the product or service; there is an obvious benefit for them and therefore little need for incentives. But why should anyone join a general access panel to answer questions on a wide variety of subjects? Do we really think they just like answering questions? Those joining to earn some money are unlikely to stop at one panel, they will sooner or later join more panels to increase earnings.

For all our panel members, whether professional or not, a decent level of pay is clear evidence that we are serious about getting carefully considered opinions. Evidence that this approach is successful can be found at surveypolice.com, where panel members can rate the panels they belong to. Our NewVista panellists are sure their opinions matter to us, they feel that we do high quality surveys that work well for them and that taking a survey with us is an enjoyable experience. 98% would recommend NewVista to a friend. I doubt such ratings would be possible if we were to pay, as some do, below the minimum wage.

I agree with you that we have to make our online panel surveys interesting, engaging and relevant for respondents, but as with any job we want done well, an appropriate level of pay is essential.

From: Angus Reid 
To: Nick Sparrow
Subject: Re: Re: Re: Re: Re: Re:Re: Online Respondents
In our experience money counts – but only for about a third of our panellists. The rest are motivated by a plethora of factors including a desire to be involved, to make a difference, and to engage with a well-known brand in polling. And nothing can kill these motivations faster than garbage surveys that dish out repetitive questions and grids. If you treat respondents like lab rats most will exit the scene and all you’ll be left with are those with an insatiable greed for more rewards.

SurveyPolice may be a good resource for professional respondents who are doing surveys everyday, but who wants to deal with a panel dominated by these types of members? I realise that many of the sample providers are facing major challenges but the marketplace is responding with the tremendous growth of bespoke panels and communities – a phenomenon that Forrester Research claims is the most important single development in market research today.

From: Nick Sparrow
To: Angus Reid 
Subject: Re: Re: Re: Re: Re: Re: Re: Re: Online Respondents
I think we may be agreeing. General, use access panels will tend to attract those in it for the money.

Bespoke panels can rely more on the interest of panellists in the specific subject area. But we should recognise that results are from the opinionated sub-section of all customers, or voters, who want to have their say.

My belief is that for all online research we need effective quality control as unsupervised panellists may, over time, become casual in the tasks we set.

If quality control means sub-standard work is rejected – and therefore logically not paid, then it follows that good quality work must be appropriately rewarded.

June | 2008