Fishing and cooking – how to make online samples fit for purpose

In a session chaired by Adam Phillips, executive editorial board member of the International Journal of Market Research, delegates heard that the appeal of cheaper, faster online research was often tempered by concerns about the validity of the results.

“Online is huge,” Phillips said, estimating that about 80 per cent of research interviews in the UK are now conducted online. But online research covers a huge range of activities, all susceptible to different kinds and degrees of error, the conference heard.

Reg Baker, executive director of Marketing Research Institute International, said the term ‘margin of error’ was widely misunderstood; what users of research needed to know – and suppliers needed to be clear about – were the assumptions that any online research model made, and what effect they had on the validity of the results.

“Samples are like medicine,” he said. “Every good sample should have a proper label with instructions about its use.”

Corrinne Moy, global director, marketing sciences at GfK, said the industry faced a similar challenge when telephone surveying first began, given that only about 70 per cent of the population had a phone. As a tool for researching people in lower socio-economic echelons, it was not fit for purpose, but for consumer research aimed at the top end of the market, the fact that some of the population wasn’t represented didn’t matter. In the same way, online research is not suitable for every circumstance, she said.

Moy said there were two phases involved in getting the most useful results from online research: fishing and cooking. Fishing meant controlling for known differences between the universe and online panellists. Cooking meant adjusting the resulting data to eliminate bias as far as possible.

Douglas Rivers, chief innovations officer at YouGov, said it was important that researchers explained to clients that there were errors inherent in survey research that went beyond sampling error, or margin of error.