ARF seeks backers for study of panel management practices

The first study cost around $1m, and the ARF hopes to raise around twice that amount to fund this latest research-on-research initiative.

ARF’s chief research officer Don Gloeckler told Research: “To some degree Foundations of Quality 1 identified the problem and Foundations of Quality 2 is going to help explain why that is – and maybe identify solutions.”

While the first study looked at things like inconsistencies between online panels and the effect of respondents being members of more than one panel, the new study will focus on some of the practices used by companies that manage panels and how these may be influencing results. This includes some of the steps taken to deal with problems identified in the first Foundations of Quality study, which may have unintended effects.

One of the most pressing points is the weighting of online samples. ComScore chairman Gian Fulgoni, who sits on the ARF’s board, said there is now a consensus that simply weighting an online sample on demographic variables “does not solve the problem”.

“There’s something beyond demographics in the type of people that you get,” he said. “So one of the things we’re hoping is that we can tease out what those other variables are. If we can come up with a way of making sure that you could take any source of respondents, and come up with a weighting scheme by asking some questions that get at the issue of whether you’ve got a fully representative group of people or they’re slightly skewed… If we can crack that, that would be huge.”

Foundations of Quality 2 will also look at the use of survey routers to direct a respondent towards any of numerous surveys they might be eligible for; the incentives offered to respondents; and the effect of the steps taken to keep out respondents who are duplicated, have taken too many surveys, or are members of too many panels.

The research will be divided into seven parts, each led by senior executives from research suppliers including YouGov, Toluna and Ipsos Interactive Services. The organisation hopes to have the research in the field this summer, with findings ready in the autumn.

The ARF used its annual Re:think conference in New York this week to call on companies to take part in the study. Research users are being asked for cash contributions, while suppliers can contribute either money or services. Several companies have already committed funds, with others expressing interest. “I think we’ve lit a spark,” said Gloeckler.

He added: “Online research is critical to the industry today and yet there’s a lot we don’t understand about how to use it effectively. We need to understand that so we can be sure that the business decisions we’re making are the right ones.”