If panels can’t be truly representative, how do we decide if they’re fit for purpose? This was the question tackled in a debate on online panel quality at Esomar Congress yesterday.
The focus of the discussion was on how to achieve consistency and replicability in panel-based research – and whether this is enough to reassure clients.
Jeff Hunter of General Mills said progress had been made in the quality of online research, but that “we still struggle a little bit” with comparability. His company, he said, takes a “reasonably pragmatic” approach and is experimenting with various solutions.
Hunter said his organisation had not had representative data for “a couple of decades” and had accepted the fact that there will always be differences between data from different sources, and there will always be unknown flaws in any sample. You can still get predictable responses and “reach good judgements”, he said.
“The vast majority of the insights that are having positive results in-market actually flow from a lot of different kinds of research that are close to consumer lives – semiotics, ethnography, observation, netnography. None of those have representative samples underlying them.”
Hunter said that the plethora of different panels and ways of reaching consumers online meant that the ‘wild west’ of online sample had still not been tamed by widely accepted best practice. Reg Baker, who chaired the discussion, agreed that “it feels like we’re constantly in catch-up mode”.
There was some agreement that representativeness is a thing of the past (Ray Poynter suggested Esomar members who claim their samples are representative should be banned), and that we need to decide in other ways whether they are ‘fit for purpose’.
But not everyone was convinced by the focus on consistency. One audience member urged the panel members not to “sell themselves short” by giving up on representativeness. Phil Garland, VP of methodology at SurveyMonkey, said: “If we chase after consistency and the world changes one day and some people like Diet Coke with Lime less, how will we know?” Hunter replied: “Sales will go down.”
“I don’t actually owe my corporation samples,” said Hunter. “I owe my corporation insights. I owe my corporation good judgements. And even though I’ve had questionable samples over the past decade, we have a pretty good history of giving people good judgements.”
1 Comment
Mike Thompson
13 years ago
I can't completely undersatnd that those who run access panels want to assign representativeness to the past. However I think we have to be more specific. Demographic rep (sorry to use the abbreviation but I will never spell it correctly a secnd time) is easy to achieve especially in an access panel. Where the latter are hopeless in in attitudinal rep which is much more difficult to measure. It is such a shame that we are destroying the best medium the industry has ever had by allying it to the worse technique the industry has ever had - the access panel. Anyway, the real issue with rep is designing a survey that attempts to ensure that samples are drawn as randomly as possible form the relevant universe - this is the inconvenience that the access panel pushers want us to forget.
Like Reply Report