If panels can’t be truly representative, how do we decide if they’re fit for purpose? This was the question tackled in a debate on online panel quality at Esomar Congress yesterday.
The focus of the discussion was on how to achieve consistency and replicability in panel-based research – and whether this is enough to reassure clients.
Jeff Hunter of General Mills said progress had been made in the quality of online research, but that “we still struggle a little bit” with comparability. His company, he said, takes a “reasonably pragmatic” approach and is experimenting with various solutions.
Hunter said his organisation had not had representative data for “a couple of decades” and had accepted the fact that there will always be differences between data from different sources, and there will always be unknown flaws in any sample. You can still get predictable responses and “reach good judgements”, he said.
“The vast majority of the insights that are having positive results in-market actually flow from a lot of different kinds of research that are close to consumer lives – semiotics, ethnography, observation, netnography. None of those have representative samples underlying them.”
Hunter said that the plethora of different panels and ways of reaching consumers online meant that the ‘wild west’ of online sample had still not been tamed by widely accepted best practice. Reg Baker, who chaired the discussion, agreed that “it feels like we’re constantly in catch-up mode”.
There was some agreement that representativeness is a thing of the past (Ray Poynter suggested Esomar members who claim their samples are representative should be banned), and that we need to decide in other ways whether they are ‘fit for purpose’.
But not everyone was convinced by the focus on consistency. One audience member urged the panel members not to “sell themselves short” by giving up on representativeness. Phil Garland, VP of methodology at SurveyMonkey, said: “If we chase after consistency and the world changes one day and some people like Diet Coke with Lime less, how will we know?” Hunter replied: “Sales will go down.”
“I don’t actually owe my corporation samples,” said Hunter. “I owe my corporation insights. I owe my corporation good judgements. And even though I’ve had questionable samples over the past decade, we have a pretty good history of giving people good judgements.”