Meet the man who wants to make surveys history
Graves, you’ll gather, is no big fan of market research. More specifically, he doesn’t go in for the concept that you should ask people questions, analyse their answers and use that information to make crucial business decisions.
So committed is he in his belief that he’s written a book about it, Consumer.ology [sic], in which he seeks to undermine once and for all the “pseudo-science” of market research. “My point in writing the book is to say that at the moment businesses, organisations and politicians are putting a lot of faith in something that is extraordinarily haphazard as a tool,” says Graves. “Is market research always wrong? Of course not. Is your horoscope always wrong? No.”
On the road to ruing
It wasn’t always like this. Graves was once a fully paid-up member of the market research community. Brand strategy and research manager at Pizza Hut, market research manager at Eagle Star. “I worked for a couple of years for a traditional market research agency, a small one, specialising at that time in financial services research. So yes, my background into research was fairly traditional,” says Graves.
What started him down the path to where he is today was a focus group he was conducting in Leeds on personal pensions. “I was with this group of men and they told me they wanted a range of funds, financial security, independent advice – all the same things that were coming through from the other groups I had done.
“At the end of the group, after I’d turned my tape recorder off and put my notebook away, one guy asked another guy: ‘So what are you going to do about getting a pension?’ And he said: ‘Well, we’ve got this guy who comes round our house who deals with all our insurance stuff so I’m going to speak to him about it.’ I realised what he was describing was the man from the Pru – he was describing an industrial insurance salesman, with no reference to financial security, no sense of what range of funds would be available, certainly not getting independent advice. And the other guys started asking for the salesman’s number. And so off I went to write my report about independent advice and financial security and the like, and off they went to do something completely different.”
“My point in writing the book is to say that at the moment, businesses, organisations and politicians are putting a lot of faith in something that is extraordinarily haphazard as a tool”
After this “initial trigger”, as he calls it, Graves started to study psychoanalysis and transactional analysis. “It dawned on me that all these people who go through a process of psychoanalysis do so because they can’t get to the bottom of their own thoughts,” says Graves. “That then opened my eyes to the extent to which we are not very good at explaining and understanding our own thoughts. Then, when you start to look at what does make a difference in terms of how people behave, there is the unconscious mind and the emotional brain that is shaping decisions we take, and we don’t have a direct connection to it. We become awful witnesses to our own behaviour, and also hopeless at identifying what it is we do and don’t want.”
This won’t come as a surprise to many researchers – it is widely accepted that what people say they’ll do is often very different from what they actually do. This explains why attitudinal data is often analysed alongside behavioural data. Graves’s question is why people bother with attitudinal data at all? In his view there is no place for it in the decision-making process. “I would put it all to one side,” he says. “I would do away with it.”
Telling tales
While Graves’s view of research is likely to enrage certain sections of the research community, it’s possible that his animosity towards asking questions of consumers will stand him in good stead with those who argue that the role of research, especially in the social media world, should be to listen to consumers and let them steer the conversation. Graves himself is certainly less opposed to the concept of listening – though it’s not without its drawbacks, he says.
He cites fashion retail chain Zara as a “great example of a company who really listen to their customers”. “In Zara stores,” he says, “one of the tasks their managers have is to listen to what their customers tell them. So if someone comes in to complain that a belt is a bit short, say, that’s a valuable piece of feedback and they as a business are geared up to channelling that information back into the marketing, product design and product manufacturing base.”
Listening in to online conversations, though, is merely “OK”. “It’s a bit like listening to what people are saying about you in the playground. You need to understand the psychology of human communication,” says Graves. “For example, if something bad happens to me as a consumer I’ve now got a story to tell and it’s going to have all the elements of story – a good guy, a bad guy, a setting, a turning point and a hero. And I’m going to tell that story because of how it will make me look and because it’s a way of conversing that I know will engage people.”
The act of telling a story actively primes others to share their own tales and before long, Graves says, “we’ve got everybody in the playground sticking the boot in”. Companies, he says, should be alert to this, and ready to go in, deal with negative sentiment and attempt to create a positive story for its customers to tell. “But the notion that what people choose to talk about is representative of what they do” – and thus contains insight to explain customer behaviour – “just doesn’t bear scrutiny to me,” says Graves.
Where do we go from here?
So if asking questions is off the table, and listening isn’t all its cracked up to be, what are companies left with? Graves advocates observational research – watching what consumers do and analysing their behaviour.
If a company is developing new products, he says, prototyping and live testing of concepts on store shelves is the way to go. This, we suggest, might be a tough sell commercially. Companies would have to get quite far along in the product development process before they realise they have a dud on their hands. Surely that creates a bigger risk of more expensive failures than a few concept testing surveys would – no matter how flawed Graves says they are.
“We’ve been seduced by this illusion that we can ask people what they think and learn from it”
He disagrees of course. “Testing new concepts with consumers is so bad as an exercise. Companies feel that they are minimising their risk by doing that research and it’s giving them confidence to go forward, but in reality they are just as likely to be ditching a really good idea at that stage,” says Graves.
“If you are sensitive to loss, as a lot of businesses are, be sensitive to the loss of just kicking out a perfectly good, perfectly marketable idea because the process you are using cannot give you any reliable sense of whether people are going to buy it.”
It’s at this point Graves points to a list of brands that have launched on gut instinct and been successful, despite having scored poorly in research tests: Red Bull, Absolut Vodka in the US, Baileys, the Heineken ‘refreshes the parts’ campaign. “Market research was all for stamping on them and throwing them out the door,” he says.
These and other so-called “market research failures” are well documented. But it is, I suggest, an unfair argument. The story of creative genius triumphing despite the warnings of customer experts makes for an engaging tale, and reflects well on the teller. If, as Graves said earlier, people choose to tell a story because of how it makes them look, it’s hardly surprising we hear this sort of story more than the one in which the humble survey led to the development of a successful product.
Researchers looking for a ‘good news’ story to tell might point to the continual growth of their industry in the 60 or 70 years since it emerged. Surely, they might say, an industry responsible for more misses than hits wouldn’t have enjoyed such a run of success.
No one would argue that the survey is a perfect tool – nor for that matter is a depth interview, or a focus group, or any other method that involves asking respondents questions. Asking questions invites focalism and encourages people to post-rationalise, neither of which are helpful in understanding the consumer’s true sub-conscious decision-making process.
Researchers are well aware of the limitations of their range of tools. What we’re left to ponder is whether the companies who pay them to wield those tools are equally knowledgeable, for Graves’s prediction of an end to the old paradigm of market research depends on those buyers.
He says: “We’ve been seduced by this illusion that we can ask people what they think and learn from it.” But there’s every possibility that companies have really been seduced by the fact that asking people is cheaper and less time-consuming for companies than the alternative methods of research – no matter how much more reliable they might be.
‘Consumer.ology, The Market Research Myth, the Truth about Consumers and the Psychology of Shopping’ by Philip Graves is published this month by Nicholas Brealey Publishing.
We hope you enjoyed this article.
Research Live is published by MRS.
The Market Research Society (MRS) exists to promote and protect the research sector, showcasing how research delivers impact for businesses and government.
Members of MRS enjoy many benefits including tailoured policy guidance, discounts on training and conferences, and access to member-only content.
For example, there's an archive of winning case studies from over a decade of MRS Awards.
Find out more about the benefits of joining MRS here.
23 Comments
Terry Purvis
15 years ago
As someone who has conducted thousands of interviews in the name of market research I fully agree with Philip Graves.
Like Reply Report
Paul Vittles
15 years ago
It's actually the same debate that has gone on for the 25 years I have been a professional researcher. How do we understand what people do, rather than what they say they do? I believe that sometimes asking questions to help understand behaviour has a role to play (in understanding and predicting behaviour), although I agree with the critics who say we waste money on surveys focusing on attitudes. I also think that much analysis of online conversations, eg on Twitter, tells us little about behaviour (in fact Twitter has gone backwards from 'what I'm doing now' to 'what I'm thinking now') but we are getting genuinely new insights through new online methods of observing and analysing behaviour - or online behaviour at least. And we have to listen to these commentators like Martin Lindstrom and Philip Graves don't we? As Maureen Lipman's character in the BT ads used to say - they've got an ology!
Like Reply Report
Allan Shaw
15 years ago
The initial "Eureka" moment here could seem slightly flawed in that Philip instantly assumes that the "guy who calls round the house and deals with all the insurance stuff" is a man from the Pru type figure who is offering only one range of products. We too have a man who calls round the house to deal with these things but he is an Independent Financial Adviser who offers us a whole range of products/services uniquely tailored to, and fully meeting, our requirements. Maybe I'm wrong but has poor research led to writing a book about how research/surveys are not going to be needed in the future?
Like Reply Report
Chase Murdock
15 years ago
Reminds me of some work Malcolm Gladwell's done on the nature of choice and consumer selection. Sometimes consumers just don't know what they want in a product. http://www.ted.com/talks/malcolm_gladwell_on_spaghetti_sauce.html
Like Reply Report
Michaela Mora
15 years ago
It is interesting that Graves' "initial trigger" took him to the path of dismissing asking questions instead of questioning if he was asking the right questions. I guess that would have been a tougher road to take for him. I have seen the birth of very successful products based on asking questions about both attitudes and behaviors. We need to know not only what people do, but why they do it. They might not always be able to formulate what they want in the future, but it is our role as market researchers to tease that out with the right questions and combinations of techniques. Unfortunately so many bad surveys have been created particularly in the era of cheap online survey tools when everybody thinks they can write one, that they have overshadowed the outcome of really good and professional research. Hopefully, as the recession eases up and research budgets come back to life again, companies will be able to invest in quality research and we can see the death of bad surveys. Michaela http://www.relevantinsights.com
Like Reply Report
Scott Taylor
15 years ago
This might be oversimplifying, but it sounds to me like this will take one existing methodology, which is admittedly flawed and replace it with another methodology, which will also be flawed.
Like Reply Report
Kathryn Korostoff
15 years ago
As someone who has done 600+ market research projects over 25 years using various techniques, I can tell you the following: 1. Surveys are not perfect, and alas are often used when other techniques would yield better insights. 2. Surveys work very well (are reliable and accurately predict behavior) in some markets/customer groups more than in others. Anyone doing research for more than 5 minutes can tell you that, including Phillip I'm sure. 3. Observational methods are not perfect, can be very misleading, and simply don't apply to many research situations. 4. In-depth interviews and focus groups have their own set of limitations--using them (or any method) inappropriately will yield misleading results. 5. The bigger issue is over-relying on any single method--there is an appalling lack of triangulation and contextual consideration in many research projects. Bottom line: no market research method is perfect, and buyer beware of any author/salesman proffering a one-size-fits-all solution.
Like Reply Report
Steven Cierpicki
15 years ago
Yes this has been a debate for a long time but I think Philips long term projection is worth considering in a world now awash with lots more real behavioural data. Traditional MR (asking what people need, want, like, might do) has a part to play in the decision making but it will be an increasingly marginalised input alongside the real/hard information. So the balanced perspective Kathyrn outlines will be even more balanced by inputs external to what MR has the ability to provide. So what would an MR industry look like if no new data was collected & researchers had to just use what data already exists?.
Like Reply Report
Steve Morantz
15 years ago
In its best manifestation, the prinicpal tool in qualitative research is the qualitative researcher, not the questions that she asks. Qualitative research is not a simple process of transferring information from one place to another and totting it all up - 2+2 = 4. this is precisely why it is different from quant. It's an exploration by a creative map maker who gives us the colours and shapes to describe the new world she finds. Perhaps the key to his views is in the apparently very limited experience that Graves had as a researcher.
Like Reply Report
Scott MacLean
15 years ago
“In 20 or 30 years,” he says, “we will look back with genuine embarrassment at what we currently do." ? Actually, to misquote Keynes (?), in 20 or 30 years, we will all be dead.
Like Reply Report
Senior Management Research User
15 years ago
A rather naive perspective from extremely limited experience, but wielding a large paintbrush. By this flawed logic, Governments might as well stop doing censuses. The biggest limitation of the research industry is that successes and strategic stories are always confidential and never shared. Very few make it to the IPA case studies or shared in media. But the hundreds of best brand stories and how they came about are not shared as clients feel it is sensitive/ confidential and might reveal their competitive advantage. and what turns this on its head is the fact that all the big brands on the digital space are also big buyers of traditional research and media. (But again, you might hear about how they benefited from it only if they share it at an IPA awards function or some case study, article is shared). But sadly instead of the success stories what gets publicity is a rather sensational driven, naive, uninformed work like this author's book.
Like Reply Report
Anon
15 years ago
I find it interesting that a number of people commenting on this article have selected one element of a relatively short interview about an entire book on which to arrive at a conclusion. I admire Research-Live for airing this discussion, but there is, inevitably, a degree of self-interest involved. I think there are two parts of this interview that, as a psychologist myself, I find particularly interesting: firstly, the following sentences: "Then, when you start to look at what does make a difference in terms of how people behave, there is the unconscious mind and the emotional brain that is shaping decisions we take, and we don’t have a direct connection to it. We become awful witnesses to our own behaviour, and also hopeless at identifying what it is we do and don’t want." Psychologically, these statements are well supported by scientific research; those people dissenting with the broad argument would, I believe, do well to focus their attention on this point. Secondly, the fundamental counter-argument is the "continual growth of their (the market research) industry"; are we to take it that an acceptable defence of something like astrology (to borrow a reference from the interview) is the extent to which the market for it is in growth? I will read and consider the full argument of the book before jumping to any conclusions. However I do think the interview, despite its Research-Industry source, raises some very reasonable challenges and fails (along with the comments to date) to provide a scientifically-based counter-argument. If market research is essentially a faith, we should at least acknowledge that, rather than pretending it's a science.
Like Reply Report
Nigel Hufton
15 years ago
While I would like to read his thought-provoking book before drawing a conclusion, I hope Graves can give better examples than the pensions one to substantiate his claims. On this basis, I would have thought the plea should be for better research, not less research? Rather than going "to write my report about independent advice and financial security and the like, and off they went to do something completely different”, perhaps he should have done the following: 1) Realised that this was an important insight and reported it back to the client (whether or not the tape had finished!), which might affect the validity of the findings. If I had been the research buyer, I would have expected as much. 2) Explored some of the responses in the research better (or used prior knowledge of the market). We know from our research that the pensions market is one in which consumer knowledge and confidence can be low, because it is a complex and infrequent decision and no-one is able to give you perfect advice (you can't know if you've made the best decision until many years later). What people say they want and what they do can differ - but they are not necessarily inconsistent – they can be understood! However, one needs to delve down to understand why there is an apparent difference and what customers underlying needs are. For instance, when his respondents said they wanted independent advice, perhaps what they were really after was good advice which they could understand and trust (i.e. not neccesarily a legal definition of independent advice). So when a respondent recommended a trusted supplier, it generated interest - in the same way as recommendation is powerful in many markets. Also, it is well-known that most people will say they want a choice of funds in research, yet most pension holders leave their funds in the default funds. Is this inconsistent? I would say not - it means that consumers like to be given choice – both at the point of sale and later – as it gives them a feeling of control and flexibility – even if they don’t ever make use of it. Good researchers employ techniques to test and interpret whether consumers mean what they say and say what they mean and don’t always take responses at face value.
Like Reply Report
Ray Poynter
15 years ago
Like the other people making comments, I shall have to read the book before I can be sure, but I am not sure that Graves is adding anything to what Mark Earls, Martin Lindstrom, and Dan Ariely have been saying for a while? Indeed (and excuse the cheek) even my book http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470710403.html has a section on why questionnaires do not work in the way we used to think they did. And without wishing to be harsh, if Graves was the sort of qual researcher who simply noted down what people said in a focus group and believed it, he would have been in a minority even then.
Like Reply Report
Will Ranner
15 years ago
I think I already look at some of the things we do with embarrassment. But Philip Graves makes a slightly off the mark argument. Other comments cover it more eloquently than I have time for, but suffice to say I agree with the likes of Kathryn Korostoff here and also others who say that things will (and must) change. Surveys have their place. Listening has its place. Observation has its place. Technology is the great enabler and the great game-changer. How we, as an industry, use technology to meld these different techniques together will be key. There are lots of examples out there already, which show people making strides in the right direction. However, we mustn't be precious about defending our market or techniques, or saying what 'counts' as market research. We must be flexible and embrace change. ISO standards have their place. So does taking one 'Tweet' to the CEO and taking immediate action.
Like Reply Report
Neil Gains
15 years ago
I agree that we should all read the book before judging, although Ray you are right that the points have been made many times before. The problem is that the research industry (on the whole) continues to ignore these issues, and continue conducting a lot of research based on models of human choice & behaviour which have no justification. As pointed out by our anonymous psychologist, there is little academic disagreement that the majority of behaviour is habitual and with a large emotional component (I would rather avoid the irrational vs rational debate). As such, it is not subject to conscious deliberation and therefore rational questioning. A large amount (perhaps the majority) of research simply ignores this. This is not to say that research is not useful, and as Kathryn points out, we should think of 'horses for courses'. In fact, I would go further and say that good research should be riding several horses and using different points of view to answer important business issues (triangulation). This includes neuroscience, analysis of behavioural data and observation (all mentioned above) as well as some more rational approaches. This is a great debate, and my only concern is that the majority of market researchers will continue to ignore some of the important messages, and focus their attention on shooting the messenger.
Like Reply Report
Brian Tarran
15 years ago
We actually have a copy of the book to give away to a reader who's willing to review the book and present a counter-argument to Graves' point of view. If anyone's interested please email: briant@researchmagazine.co.uk ------ In the meantime, I'd like to thank you all for your contributions so far. There's been a lot of attention paid to the focus group example (Graves' 'initial trigger'), but perhaps others would like to address some other points he makes: about online buzz monitoring, say, or the 'flaws' in the concept testing process?
Like Reply Report
Di Tunney
15 years ago
I like Terry, have conducted thousands of market research interviews and completely agree with Philip. In the past we did the best we could with what we had but most widely used market research techniques are well past their 'best before' date There are now a number of 'tried and tested' approaches that enable us to decode what is driving consumer behaviour at and unconscious level and that thankfully keep 'researcher influence' out of the equation. I am somewhat surprised and disappointed at the rather defensive position being taken in some of the comments to this post. Surely being 'open-minded' and prepared to see things from a different perspective are two of the key qualities we need to have to be good researchers?
Like Reply Report
Anon
15 years ago
Thought some of you might be interested in the interview with the psychology and behavioural economics professor Dan Ariely on the Esomar site: http://www.esomar.org/uploads/rw/2010.09/Research-World-September-2010-Connecting-the-dots.pdf His answer to the last question looks to draw very similar conclusion to Philip's. I can't imagine many people will be taking up Brain's offer: surely all of us very balanced research types will be unable to review something knowing that the objective is to "present a counter-argument". We would want to review it to, err, review it, wouldn't we?
Like Reply Report
Greg Timpany
15 years ago
I have to concur with Kathryn and those advocating a balanced approach. There is no silver bullet here, each technique has its limitations thus we need to rely on a broader slate of data and collection methods. For those pushing strictly transactional data, remember any system is capable of having flaws. Greg Timpany http://www.linkedin.com/in/gregtimpany/
Like Reply Report
Jon Anderson
15 years ago
I'd offer a reminder that virtually all marketing research is biased to some degree, save for the rare project that only uses perfectly unobtrusive measurements. Our challenge as researchers is to identify all potential sources of bias, manage the project design so as to reduce these as much as possible and then take the remaining potential sources of bias into account when analizing the results. If we follow those principles, that should keep us from overpromising, oversimplifying, creating false expectations, etc. Failure to follow these simple (but not always easy) requirements is what leads to producing junk "science" masquerading as market research. One of my favorite (bad) examples is Frank Luntz's TV "focus groups." What is up with that? It's like research malpractice. Come on, Frank, call it something else.
Like Reply Report
edward
15 years ago
I find it's easy to attack research - it's been going on for ages, unreliable methods, people not saying what they really think or saying stuff and then doing something else. Lets not forget that for every study where research was "wrong" there are perhaps 10 studies where research gave a solid steer in terms of a direction or an action plan. but if you've got a book to promote......
Like Reply Report
Sean Jordan
12 years ago
I'm a little late to the party to comment on this, but since I didn't see anyone mention it... Does it strike anyone else as ironic that Graves uses an anecdotal "aha moment" to dismiss an entire spectrum of research as pseudoscience? The reality is that a lot of marketing research is not done well, and that's often because the "need to know" is greater than the desire to have information that's validated and reliable. Drawing definitive research conclusions from a single method of research, such as focus groups, is not scientific, and it ignores many steps in the research process that are necessary to ensure that the information being collected is valid and reliable. Research done properly (and scientifically) is a process, and it requires not only reviewing secondary data, but also constantly looking for new angles on primary research through further study. The compromise that is often made is that a single study is performed in the interests of time or money, but even in these situations, a professional researcher who is well-trained in scientific methods can build in systems for validation. What Graves is pointing out is a symptom of a real problem that is smoothing itself out as research professionals have begun to take better charge of their own field. Too often, those who call themselves professional "researchers" are not trained in proper research methods and are merely analysts or focus group moderators who have been elevated to senior research positions. But this is changing; professionals can obtain certification through the MRA's PRC program, and there are also several top-notch graduate programs for marketing researchers. (I'm am alumnus of one of those, in fact.) There certainly is some noise in the MR industry, and I don't think any professional certified researcher would deny that. Oversimplified research that doesn't take the psychology of consumer behavior into account can be dangerous indeed when it's being used to support million-dollar marketing decisions. But Graves really seems to be throwing the baby out with the bathwater.
Like Reply Report