NEWS9 August 2010

Online brand impact studies a damp squib? IAB-funded probe says so

North America

US— A two-year investigation into the state of play in internet ad effectiveness (IAE) research has delivered a worrying conclusion: “One cannot be confident whether the findings of most IAE studies are right or wrong,” according to report author Paul Lavrakas.

Low response rates – sometimes less than 1% – threaten the external validity of IAE studies, while internal validity is at risk from the “near exclusive use of quasi-experimental research designs rather than classic experimental designs”, said Lavrakas, Nielsen’s former chief research methodologist.

He also highlights “a lack of valid empirical evidence” that the statistical weighting adjustments used in most IAE studies adequately correct for the biasing effects of the methodological limitations.

But most damning is the fact that researchers are aware of the shortcomings of current methods and have other methods they could employ to improve the validity of the research – its just that the online advertising industry marketplace “has neither demanded nor been willing to fund the type of IAE studies that can generate findings known to have strong internal validity and external validity”, said Lavrakas.

“It appears that as currently perceived by most who fund IAE studies, the cost and complexity of funding studies known to be valid versus the benefits of doing this does not support the use of the more rigorous methods,” he said.

Lavrakas recommends a series of studies be commissioned “to help resolve the uncertainties that currently exist”. One project could compare the results of classic experimental design against quasi-experimental design; another could investigate the size and nature of nonresponse bias.

The investigation, commissioned by the Interactive Advertising Bureau (IAB), focused on brand impact studies which use site intercept methods (and sometimes online panels) to survey web users who have been exposed to an ad campaign, with results matched against surveys of those who have not been exposed.

Sherrill Mane (pictured), the IAB’s senior vice president of industry services, said: “If the phrase ‘marketing science’ is to have any meaning, participants in the ecosystem must demand that their vendors employ rigorous, tested research methodologies, even if doing so costs more.”

The IAB is following up Lavrakas’s work with cross-industry task force and the creation of a set of US best practices for online brand impact studies.

The report, ‘An Evaluation of Methods Used to Assess the Effectiveness of Advertising on the Internet’, can be downloaded here.