Testing times

Research: Do you like market research?
JW: Absolutely in theory but very rarely in practice. It should be very useful, but there are structural and methodological reasons why it often isn’t. I don’t think it’s because of a lack of willing from people in the research industry, I think it’s the interaction between research and large corporations that’s essentially problematic.

I would love to see much more scientific and particularly psychological rigour applied to the way research is carried out. We now know a hell of a lot more about how the human mind works and how advertising works than we did when I started in this industry in the late 1990s, and I don’t see that reflected in the way that we test advertising. There is a massive wealth of literature which shows that asking people why they’ve done things or what they think they’re going to do is not an accurate method of predicting people’s behaviour.

Most researchers would say they are aware of that. But you don’t see it reflected in the way research is done?
If we take a very standard pre-testing measure, like brand appeal, it’s primarily measured by asking people, ‘Do you think after seeing this ad that you are more likely to buy this product?’ Many clients would use that as the single measure for making a choice about whether an advert works – and it’s absolutely based on introspection. It completely discounts a wealth of literature that’s out there such as Binet and Field’s Marketing in the Era of Accountability, Robert Heath’s work, Andrew Ehrenberg’s work, all of which shows that many of the effects of advertising in many cases can be due to effects that people aren’t conscious of.

If research as a proportion of marketing spend has grown, the question that the industry has to ask itself is: is their clients’ advertising actually becoming any more effective? We’re in an era when what we do is very strongly questioned for all sorts of reasons, so it’s crucial for everyone in marketing to better engage with the robust truths about how what we’re doing works. There are lots of convenient fictions out there which do nobody any good, and make it more likely that we as an industry produce stuff that consumers out there find irritating, that waste’s everybody’s money, and that makes marketing less sustainable as an activity. Business likes to look at itself as faster and more responsive than government, but you’re in a situation now where you see the Conservative Party more engaged in things like behavioural economics than the marketing industry is, and I just find that utterly laughable.

Has the way research is used got worse during your time in advertising?
I think I have seen the reliance on pre-testing get heavier. It’s a seductive thing to be able to justify why you ran an ad that didn’t work very well, or if you’re a global brand director in London trying to make sure your guy in Vietnam is producing acceptable advertising. That’s why it’s so prevalent, and for some organisations that’s exactly what they need. But research that is optimised to help marketers make decisions is not optimised to maximise creativity and effectiveness. They are different objectives.

I also think it has become a less intellectually rigorous exercise. You can go for a year with some clients and they will not mention anything other than a secondary measure. A lot of the things we talk about in marketing, even the idea of a brand, are useful theoretical constructs for how what we do affects people’s behaviour, but when you start treating those constructs as ends in themselves, that’s when what you do becomes unreal. It’s research agencies who are in a position to tell us when we’re wasting our time.

And they’re not living up to that?
I don’t think so, no. One observation that’s growing is that in tracking studies, individual brand measures don’t tend to move independently, they tend to move en masse. This is a great example of the kind of place where we could trim the fat a bit, and then more money could be invested elsewhere.

What role do the relationships between researchers and ad agencies play?
Advertising agencies certainly don’t help themselves in their dealings with research agencies. Most ad agencies walk into that research debrief meeting with the feeling that the research agency is about to destroy their ad. The reason there’s a gulf between the two is that most pre-testing methodologies seek to make sure the client is placing their ad in the right half of the bell curve [showing the effectiveness of different ads], and they probably do achieve that: they get rid of the disasters. However I would also maintain – and this is much harder to prove – that those methodologies make it very hard for you to make a truly excellent ad.

So what’s the right way to go about it?
I think things have to be assessed against how they’re designed to work. Hall & Partners, from what I understand about their approach, will start from, ‘What’s the advertising strategy here?’ and will then tailor their methodology to that. Another company that I think have a very interesting approach is BrainJuicer, who avoid that problem of explanation because they ask consumers for a straight-out emotional response.

Another thing I think we need more of is post-testing – did the campaign work? Were I in charge of an advertising research agency, I’d be thinking about how to shift my clients from investing in pre-testing to investing in post-testing and econometrics. And digital media offers us a great opportunity to be faster about how we do things like post-testing, so there’s loads of hope.

But clients need to know whether something’s going to work now, not after the fact.
If you’re the advertising agency, you’re saying, we as a company have x years of experience of producing advertising, and based on that we recommend to you what we think is going to work. To then have something which, frankly, can seem reductive and methodologically dubious held up as being superior to that expertise is very bruising to the ego and, I think, not always supported by the evidence.

Were I a client, I wouldn’t invest in pre-testing. I would far rather have, for example, a payment-by-results system with my agency. I’d invest my money in testing whether stuff had worked and in doing econometrics, and then reap the efficiencies in how much money I paid to the agency. If they’ve done a good job, they get more cash, I get better results. If they’ve not done a good job, they get less.

To me that’s a better use of the client’s budget than testing something which is not what you’re actually going to make, in a situation which is not analogous to the one in which the thing you’re making is going to be, and using a methodology that doesn’t accurately reflect how the thing you’re making is going to work.

You can’t give up on pre-testing, though, can you?
The question to ask is, how do you do it in a cost-effective manner? I’ve worked with a client who has tested a regional ad 27 times – for the cost of that you could have made the thing twice over. What a research agency is well placed to do is to find quicker and cheaper ways to do pre-testing, because to me the cost-benefit equation is not there. I think the way to success is to work back from what the business effects are, to work out what are good predictors of those. The truth about many of the legacy methodologies out there is that they’re based on what people thought about marketing back in 1960 or 1970, and because there’s been a desire to keep using the norms and to build on the way things are done, we’ve been stuck with them.

The biggest name in pre-testing is Millward Brown. What do you make of their approaches?
If I were in a situation with a client who had a clear product advantage and a rational reason to choose that product, I would be happy to test it with Millward Brown. In other contexts I’m less sure. What they have that I would love to get my hands on is an incredible dataset. But were I in their position I’d be worried, because the media environment is changing, and one has to question how relevant all of that data still is.

Is there a way around the tension between ad agency creatives and research agencies, or is it inevitable?
Ad agencies are less bothered about being told whether or not their work is good or bad; what they are bothered about is being told how to fix it. The integrity of something as a piece of communication is quite a delicate thing, and to have someone come in and tell you, ‘If you moved this scene from here to here then your ad would be better’ is a very difficult thing to swallow – and quite often wrong, because I don’t think, by and large, the people who are writing those research debriefs are people who have made many successful pieces of advertising in their lives.

But isn’t it a cop-out to just tell researchers, ‘It’s all terribly complicated and ultimately we know best’?
No, I don’t think it is. If researchers come to an ad agency and say, ‘People love your ad, the problem is that at the end they can’t remember what the brand was,’ the ad agency is going to go away and try and come up with a solution that preserves what’s entertaining and interesting about the ad and the bits of it that are on-strategy – but we want to be in a position to prescribe that solution.

The views expressed are Jacob Wright’s own and not necessarily those of Mother