FEATURE1 November 2010

Ask a silly question


Philip Graves says in his new book, Consumer.ology, that market research is barking up the wrong tree in its preoccupation with asking people their opinions. Two voices from the industry offer their views on the book.

“In 20 or 30 years,” says Philip Graves, “we will look back with genuine embarrassment at what we currently do. We’ll look back on this period of market research – where we’ve been seduced by this illusion that we can ask pepole what they think and learn from it – and we’ll try to erase it from history”.

He’s set out his view of the industry in full in his new book, Consumer.ology. Click here to read Brian Tarran’s full interview with Graves. Below, two figures from the industry offer their reactions to the book.


Jeffrey Henning on Consumer.ology

The founder of Vovici says pointing out the shortcomings of surveys and focus groups isn’t enough to kill of market research


In Consumer.ology Philip Graves makes the case that belief in market research is as misplaced as belief in numerology or astrology. Businesspeople are afraid to make decisions, and rather than turn to their daily horoscope they conduct surveys to foretell their businesses’ fortunes. All the old tropes condemning research are here: from Henry Ford’s ‘faster horse’, to Steve Job’s “We do no market research”, to at least three different retellings of the research behind New Coke.

“Graves falls into the familiar trap of being in love with one or two methodologies rather than treating market research as a toolbox of many different methods, each with its own role to play”

The factual errors start on page one, with sentence one: “Market research emerged during the media and advertising boom of the 1950s,” Graves writes, apparently oblivious to the fact that Curtis Publishing created the world’s first formal market research department circa 1911 to help advertisers understand the readers of the Saturday Evening Post. Graves then imaginatively reconstructs the thoughts of these market researchers of the ’50s, before emphasising to the reader that belief in market research can be wrong because belief in other things can be wrong, citing the use of astrology by “Nancy Regan” (sic).

If I were as thoughtless as the author believes consumers are, I would have at this point discarded the book for reasons inexplicable to me. But instead I channelled my inner qualitative researcher and read on, trying to understand how Graves’ world view might be true.

As near as I can construct it, when Graves says “market research” he means “surveys (and focus groups)” and nothing else. In the subtitle of Consumer.ology, “The Market Research Myth, the Truth about Consumers and the Psychology of Shopping,” he is really saying “The Survey Research Myth”. He loves consumer ethnographics, but doesn’t seem to know or use the word ethnography. He also loves ad testing and product trials, but apparently those aren’t ‘market research’ techniques either.

His thesis, in short, is that surveys are almost always bad, focus groups are always bad, and ethnographics, ad tests and product trials are the only truly effective techniques. He digs a shallow grave for survey research by pointing out 13 potential problems with surveys from the standpoint of consumer behaviour, when he could have buried it six feet under if he had referred to some of the research behind the concept of Total Survey Error.

Yes, many surveys are bad. In fact, in some ways, they suffer from more problems than Graves is aware of. And yet, of course, surveys are a technique of proven power, but he dismisses any predictive validity for surveys, even for political polling, despite its long track record of accurately forecasting the behaviour of voters. Graves ignores the important role that surveys play in business-to-business research, where many stakeholders have a say in decisions and where decisions are, if not more rational, certainly less impulsive than in the consumer world. To Graves’ mind, if surveys are ever right they are right by coincidence, like that occasionally accurate horoscope.

Graves falls into the familiar trap of being in love with one or two methodologies rather than treating market research as a toolbox of many different methods, each with its own particular strengths and weaknesses, each with its own role to play. For instance, he sings the praises of behavioural research, especially online, yet researchers will tell you they need surveys and in-depth interviews in order to learn the goals of visitors to such sites. Following what site visitors do is not enough if you can’t figure out what they want to do. True consumer understanding comes from combining the results from multiple research methods, appropriately applied. As Graves points out, consumers find it hard to understand themselves, and it’s even harder for researchers.

If you need to talk a stakeholder out of doing a survey and into doing ethnographic research or product trials, Consumer.ology is the book for you. If you were not aware of the recent science into consumer thinking, buy Predictably Irrational or Freakonomics instead. If you do buy Consumer.ology I look forward to your post-rationalisation of the purchase.


Neil Gains on Consumer.ology

Ditching the old methods is no better than ignoring the new ones, says the founder of Asian research agency Tapestry Works


Consumer.ology will be seen as a threat to market research, and rightly so. I am pleased to see a book devoted to recent discoveries in social and medical science including neuroscience, psychology and behavioural economics, and their implications for our industry. Market researchers need to take Graves’ arguments seriously.

“Graves says we must keep in mind that behavioural data is more valid than opinion and that context, question-framing and time frame are all relevant. I believe good researchers already appreciate these issues”

But one wonders whether in the process of writing the book, Graves himself has fallen victim to confirmation bias. He compares the research industry to astrology, but fails to offer any scientifically validated alternatives. He favours observation, a vital, and still underused, research tool. However, market research is more than observational research, and it is unrealistic to fully test-market new products as the author suggests. If anything the trend in business is towards earlier, smaller-scale feedback.

Some of the book’s examples are distracting and unhelpful. The New Coke fiasco may have used approaches with serious limitations, but the main lesson is surely that businesses need to ask the right question, focusing on business-based metrics rather than spurious attempts to outsmart Pepsi. The book lacks positive examples of where research has worked, which could have provided deeper and more balanced analysis of the benefits of alternative approaches. For example, despite many flaws, political polling and new product forecasting are often surprisingly accurate, given the complex environments in which they are used to make predictions. And if research is more often wrong than right, the author fails to explain how businesses justify using it.

The book’s key ideas are, however, striking and important. The vast majority of human behaviour is not conscious and rational, and most of the time we are unaware of what we do and why we do it. The implication of this is simple: it is difficult (or impossible) to understand and interpret the vast majority of consumer behaviour through direct questioning. Researchers know this as the ‘say/do’ gap, but perhaps fail to acknowledge just how big that gap is.

When market researchers ask consumers to ‘explain’ their behaviour, they are asking their conscious minds, which had no control or understanding of what happened, but will seek to post-rationalise decisions in a way that allows them to maintain a positive view of themselves.
Direct questions should be avoided, Graves argues, as they are artificial and interfere with our perception. He also makes serious, although familiar, criticisms of group discussions, arguing that our tendency to copy others means they often reach an artificial consensus.

Graves says we must keep in mind that behavioural data is more valid than opinion; that context, question-framing and time frame are all relevant; and that covert research is more valid than overt research.

I believe good researchers already appreciate these issues. My reading of the book reinforces the importance of looking at research problems from multiple perspectives. There is no single approach that avoids all these concerns in a practical and ethical way.

As flawed as many research techniques are, they do offer insights if their limitations are understood and they are integrated with other knowledge. As Graves says, “we are awful witnesses to our own behaviour”. The best remedy for that problem is to rely on several witnesses.

Consumer.ology by Philip Graves is out now, published by Nicholas Brealey Publishing


14 years ago

Very well put Jeffrey! Professional researchers with experience know that there are not perfect research methods. They all have pluses and minuses. They key is to know when to use them, be aware of of their shortcomings and combine them with others to get a comprehensive view of the issue. When so called "researchers" like Grave start ditching the whole MR field, that tells me they don't know much about it and shouldn't be taken seriously.

Like Report

14 years ago

I believe that the responses you published elegantly encapsulate the state of the market research industry. On the one hand are the researchers like Neil Gains, who are concerned about how recent discoveries in psychology and neuroscience can be reconciled with much of what passes for accepted wisdom in market research. For researchers like Neil, while he may disagree with some of my conclusions, I believe Consumer.ology will serve as a constructive challenge to how he approaches his projects. On the other hand we have Jeffrey Henning. Henning believes that opinion polls have a long track record of forecasting the behaviour of voters; which is true, except when they’ve been wrong. In 1992 virtually all the polls failed to predict the UK election result correctly. In the last general election, Angus Reid Public Opinion’s final poll put the Lib Dems in second place with 29% of the vote – they received 23% and came third. Typically, after an instance of polling failure, the pollsters invent a new term or propose a new measurement criteria to account for what was unexplained; ‘Shy Tory Voters’ in 1992, ‘Soft Support’ for Lib Dems last time. This appears to solve the problem, until it doesn’t. Researchers like Henning should also look beyond the times when research appears to confirm reality and question what we can learn when it evidently doesn’t. Typically organisations conduct just one survey, but the differences that emerge when two are conducted independently raise questions that are problematic for companies that sell surveys or survey software. It is perhaps unsurprising that he didn’t address one such example referenced in Consumer.ology that showed a difference in support for an initiative of sixty (60) percentage points. Both Gains and Henning share one counter-argument that I believe is the by-product of an understandably self-interested perspective; the notion that the best remedy to the problems science is raising about people’s ability to understand themselves is using “several witnesses”: multiple bad witnesses are not an adequate solution if there is no inherent appreciation of the potential fallibility of some of those witnesses. This is something that the legal profession has already discovered: one US study for the journal Law and Human Behaviour found that false eye witness testimony contributed to more than three quarters of wrongful convictions (that were accurately resolved using DNA evidence). What people say can sound extremely compelling and can even feel accurate to the ‘witness’, but that doesn’t make it true. I believe the credibility of market research will be enhanced when there is a much clearer appreciation of those situations where asking people questions simply can’t help. Yours sincerely, Philip Graves

Like Report

14 years ago

Trying to refute an argument based on “self interest” ... not often a great idea, especially when one is talking about motivations! Consider... 1. I am sure Philip cannot really be proselytising behavioural approaches as inherently superior to all other forms of consumer information in cases. That would be naive. 2. But when you are selling a book (or indeed anything), it is important to have a very clear and simple sales position: hence “The Market Research Myth” will always get more attention than the thoroughly well balanced and inarguable “Why it is important to understand peoples’ real behaviour, and the things they cannot articulate, as well as what people can say, and why therefore it is important to use a holistic combination of tools that are fit for purpose for different consumer insight challenges”. 3. Since I am certain Philip is not naive, and equally certain that he is in the self-interested business of selling a book, we need to interpret his actions in the above context. Based purely on an observation of the author’s behaviour on this site (and in this context, we may take what is written as the behaviour in question), we might conclude that he’s emphasising a deliberately provocative position in order to encourage people to buy the book. I may be wrong, and Philip would be within his rights to deny this imputed motivation. But wouldn't any such denial be invalidated by the axioms of his own argument?

Like Report

14 years ago

Yeah I basically agree with what the previous person just said.

Like Report

14 years ago

I've only belatedly discovered Andy's entertaining post and thought it was worth responding on his points about behavioural approaches, false dichotomies and self-interest. It is apparent that people and / or the media like to distil issues down to false dichotomies. Perhaps because of the limits of attention or space, things tend to be polarised at the outset. That said, in one very important sense, I do believe that covertly observed behavioural approaches are inherently superior to anything obtained via interrogative market research. The former provides factual data: what you are seeing is what really happens. The same is most certainly not always the case when people are answering questions in market research. "The Market Research Myth" is a simplification; it should perhaps more fittingly be the market research "myths", because there are a number of assumptions behind the use of most market research that are demonstrably untrue. In your elongated alternative version of the book's subtitle when you say "... as well as what people say..." you reveal yourself as having succumbed to one of these myths: what people say frequently isn't a valid reference point and allowing it into the mix can be a costly mistake. As for my interest in book sales; I didn't write the book to be provocative and my publisher didn't encourage me to be extreme for the benefit of making sales. I'm not alone in believing that a lot of the money spent on market research is wasted. You may consider me naive, just as I may consider you blinkered by your own beliefs. You may think my motivation is selling books; I may think yours is protecting your job. We may both be wrong and, in any case, it's probably not as black and white as all that. I'm not surprised that people with an interest in selling market research react negatively to the argument I have put forward, I fully expected this reaction (although I'm gratified by comments from people who do earn their livelihood from market research and say things like "I should hate this book" but go on to conclude "I love this book and recommend it to anyone conducting market research." I wouldn't expect a devout Christian to read Richard Dawkins book "The God Delusion" and abandon his faith, rather I would expect him to cling to any aspect of an argument he felt he could challenge and overlook the totality of the evidence being presented. Similarly, I wouldn't expect Consumer.ology to change the views of people who fundamentally believe that what they sell clients, in the form of surveys and focus groups, is valid and valuable (although I would have expected more of them to go about finding a way to refute the science that shows why what they do doesn't work). Fortunately (for me) I've found that there are enough research sceptics willing to explore the issues around market research in their totality by reading my book and by considering the references that substantiate my argument, to have made writing it worthwhile. So I'm comfortable with my own motivations and, within the limits of what can be said in a few hundred words rather than 65,000, the position that I've been portrayed as adopting. If that's a denial that you consider invalidated by the axioms of my own argument then either you're correct, and that argument's efficacy is vindicated, or else you've let your own beliefs and biases cloud your interpretation of what you've observed (something I warn against in the book).

Like Report

14 years ago

A couple of quick points to rejoin the debate (I thought it had gone quiet). When I say "multiple witnesses" I don't mean that three researchers ask a respondent rather than one. Rather we should seek to understand the problem from different perspectives: behaviour is one of those, but culture and society is another, as well as psychological motivation. And there lies the bigger issue. Covert observation may be 'pure', but it doesn't answer all the important questions that any business should be seeking to answer. Although it's great to know 'what?' the behaviour is, covert observation cannot fully answer the 'why?. To understand why, we need to interact and engage with consumers. We can argue about how realistic it is to understand the why - I believe it is possible, perhaps Philip doesn't. However, unless you know the why, then knowing what has limited value, as you won't be able to change anything in the real world.

Like Report

14 years ago

A very interesting debate where it seems all parties are somewhat correct, but more importantly, you are not all listening to each other. Philip Graves, if I wanted to know whether or not you were male, there would be many ways to go about it. I could ask you the question, "Are you male?" and give you the two response options "Yes" or "No". I could take a blood sample and bring it to a lab for DNA analysis. I could simply make an assumption based on visual appearance. All of these things would provide me with similar, but yet very different types of results. The methods I use in research depend on the type of results I'm actually looking for. There is never a "one-size-fits-all" tool in market research and sometimes it's easier to understand something by just asking (behaviors too). I would also like to point out that all of the market research "errors" that Philip is bringing up could happen no matter what type of research you do and no matter what tools you use. I would list all of the errors that can occur while conducting any type of research, but I'm going to assume for now that we're all very knowledgeable in that area. Jeffrey Henning, although I also support your argument and think you made some great points, I have to agree with the generalized point that Philip made about change. If my children become market researchers, they will indeed look back at what I did and laugh, but not because of the research approaches I used or conclusions I came to. I look back at what my father did in design, and it is brilliant, but the tools he used were archaic. Technological advances will force us to do better and quicker research and to do it differently. You guys may have inspired me to write a blog post or a book ...

Like Report