Listen carefully

The IJMR (International Journal of Market Research) Research Methods Forum, held in London yesterday, was billed by keynote speaker Phyllis Macfarlane as the event “where, if you try to suggest that maybe the Emperor’s new clothes are not quite covering his whole body, you will not be shot down in flames and dismissed as a dinosaur”.

The august surroundings of the Royal Society seemed fitting for such an event – with artefacts recalling the likes of Newton and Darwin dotted around to remind us that this is a place of science, not fashion or fads.

The suspect clothes to which Macfarlane was referring are, of course, those being produced by the proponents of online panel and social media research – the big areas of focus at other industry events this year. The US market, she said, made the move to online “with alacrity”, but now they wonder if they made the right move, and they don’t always ask the right questions about methodology. “It’s almost a conspiracy that we don’t talk about the fact our samples are no longer representative,” she said.

But while the tale of the naked Emperor is certainly relevant to the research industry, so is that of King Canute, and the fact remains that these methods are fast, cheap and evidently good enough for the increasing numbers of buyers opting to use them. Take Coca-Cola’s Stan Sthanunathan, who last month told Research: “If somebody wants to question the representativeness of online panel, I can’t really argue with them. But don’t raise a problem if you don’t have a solution or at least alternatives.” Sthanunathan is unapologetic in advocating a focus on replicability and consistency rather than representativity.

Macfarlane remains concerned about the continuing rise of methods that may not be all they seem, and suggested that a juicy lawsuit from a client who has made a bad decision based on unrepresentative data might be just what we need to shake things up. “I would like to think there’d be a flight to quality,” she said, “but I wouldn’t bet any shares on it.”

The final slide of her presentation added a note of caution to the event’s theme of ‘Stop asking, start listening’. It said: ‘The survey is not dead.’

The day’s other speakers showed how the idea of ‘listening’ can be applied beyond the obvious areas of ethnography and social media monitoring. Paul Edwards of TNS-RI considered ways to apply buzz tracking techniques to ad testing, turning the ‘test then launch’ model into ‘launch then test’, and effectively putting ads in a perpetual beta phase.

Nick Sparrow of ICM spoke of his firm’s E-Delphi polling technique, which uses open questions about broad topics, and revisits respondents several times over a few days, with the ideas that emerge from each stage feeding into the content of the next. The technique is somewhere between a poll and a mini online community, and has managed to produce useful policy suggestions on issues that are too murky or complex to tackle in a simple poll.

Ed Keller of word-of-mouth specialist the Keller Fay Group emphasised that the vast majority of WOM communication still takes place face-to-face – not online – and warned that what is discussed in blogs and forums does not necessarily mirror what is said in the ‘real’ world. The lists of the top ten brands mentioned in on and offline conversations only have two brands in common, he said.

Adam Phillips, who chairs Esomar’s professional standards committee, highlighted how regulators and lawmakers are struggling to keep up with emerging technologies for observing people’s behaviour – especially online – and warned companies to avoid dubious practices “which are not yet technically illegal”.

But one of the best suggestions for how companies can apply the lesson of listening came in a discussion at the end of the day on the importance of research practitioners listening to one another. Various commentators decried the reluctance of competing agencies to discuss their methods with each other, arguing that fears of a competitor being able to steal your techniques and apply them with equal success were exaggerated. “If you liberate information you get creativity,” said GfK NOP’s Mike Cooke. “If you build walls around information you get nothing – except walls.”