FEATURE16 July 2009

Truth, lies and market research


One of the key priorities of our trade is to design research in a way that elicits honest answers from people, says Mike Jaxa-Chamiec of FreshMinds. But even with sound design and the best of intentions, market research can fail.

One of the key priorities, if not duties, of our trade is to design research in a way that elicits honest answers from people. But even with sound design and the best of intentions on the part of researchers and respondents, market research can fail.

Getting a thumbs-up from research but a thumbs-down in the marketplace is a nightmare for any project. It might happen as a result of sampling or interview error, or because respondents are bored or taking part for the wrong reasons. But it might also have to do with psychological processes in participants’ minds that cause them to respond in a way that misrepresents their true opinions and behaviour.

The psychological processes that underpin respondent error can broadly be split into group and individual processes. Group processes are more relevant to qualitative research methodologies where participants are exposed to others who could shape their behaviour. Individual processes, however, could crop up in any sort of research and both, to varying degrees, can be at play simultaneously during the same project.

The power of the herd
Let’s take the group processes first. Focus groups are a perfect example. Very often decisions made following focus groups are worth millions, and they present possibly the biggest minefield for respondent bias.

Imagine you are launching a new product, an MP3 player for example, and you are running groups to select the best one. Throughout the session your skilled moderator has got participants to discuss the pros and cons of six models, and the group settles on two preferred options. But when at the end of the session the moderator invites participants to take one of the specimens home, seven out of eight of them opt for one of the players that didn’t make the shortlist.

These things do happen, and social psychology offers numerous potential ways to explain why:

  • Herding – people unwittingly follow others. This could mean that, irrespective of what the product might have been like, the first two people may have picked the same thing, thereby prompting a chain reaction that overtakes the capacity for individual rational evaluation.
  • Normative social influence – people behave in a certain way to fit in. This might suggest that participants had not really clicked with any of the product features and that any benefits were insufficient to avoid being overturned by rudimentary group dynamics.
  • Informational social influence – people struggle to make independent judgements and instead look to others for validation. In this case, they may not have fully comprehended the benefits of the propositions or differences between them and renounced their own opinion in favour of others’.

“Participants’ responses can be affected by a range of factors during the research process, and people might be pretty bad at predicting their own behaviour, even if they are convinced of their ability to do so”

It’s important to take two things from the above scenario: participants’ responses can be affected by a range of factors during the research process, and people might be pretty bad at predicting their own behaviour (or that of others), even if they are convinced of their ability to do so.

And it can get even more complicated. Research in the 1990s showed that any group processes are amplified proportionally to the perceived importance of the task at hand. So in our scenario, participants’ awareness that someone would be using their input to make decisions about a product launch could have affected their desire to give the right answer rather than express their honest opinion.

Lonely souls
Individual research participants can be equally vulnerable to bias. Inevitably, part of this has to do with the back-end of research design. For instance, the ‘inability error’ – which causes respondents not to answer a question or answer it untruthfully, often has to do with inappropriate sampling or question design. But the fact that many respondents choose to answer a question they don’t know the answer to reveals a more internal biasing process often referred to as the ‘best light’ phenomenon.

Imagine a survey asking respondents which brands best represent them. The truthful answers might be things like Tesco, the Daily Mail and Vauxhall. But many people will opt for more aspirational brands instead: the Rolexes, Jaguars and Armanis of this world become standard responses even if they have precious little connection with the respondents’ day-to-day lives. The interviewee attempts to project a positive image of themselves and in doing so biases their answers, perhaps because they are trying to make the interviewer like them, or give their self-esteem a lift.

Truth serum
Good design, testing and execution need to be observed at all times to avoid these pitfalls. There are also a few further steps you can take. In qualitative research, it is advisable to:

  • consider triangulating methodologies. Observation studies or ethnographic approaches can go a long way towards neutralising respondent bias;
  • make the most of non-verbal communication. A good qualitative researcher is able to pick up non-verbal cues from research participants and cross-examine those against verbal responses;
  • engage participants before and after the group. This can be done online using tools that allow participants extra familiarisation and expression, and enable the researcher to gain additional insight into group members and their contributions.

There are two particular recent examples of successful incorporation of psychological thought into quantitative market research. BrainJuicer has put together an online concept testing methodology called Predictive Markets, built on Galton and Surowiecki’s crowd behaviour studies. By aggregating individual responses to tested concepts, they sidestep the challenge of ‘groupthink’, whereby physical groups end up supporting bad ideas. This approach is a compelling marriage of the benefits of crowd opinions with the ability to eliminate the shortcomings that stem from group processes.

“An understanding of the psychology behind why respondents give the answers they do should be an integral part of the research process”

Another technique called Implicit Attitude Testing (IAT) is widely used by a working group at Harvard University. It was recently applied in the UK to a project looking at attitudes to facial disfigurement. Rather than asking participants to self-rate their own prejudice, this online approach tests the associations between words and images that people make and measures the time taken to respond to different pairings of stimuli. By focusing on subconscious rather than conscious psychological processes, IAT is capable of eliciting deep-laid attitudes to social phenomena that are very susceptible to normative or ‘best light’ respondent bias.

These examples show how an understanding of the psychology behind why respondents give the answers they do can be an integral part of the research process, rather than something that is just bolted on. Market research isn’t a discipline that exists in isolation. Good research should bring together a wide range of skills and disciplines, and this one should be a key consideration for agencies if clients are to be sure the decisions they make are the right ones.


15 years ago

Very interesting truth. Now imagine this matter when you analyze under a different data collection modes perspective. There are many studies comparing the differences between studies made over Web, telephone interviews and even qualitative approaches. I was wondering if we have some of these studies under the perspective of this article, attentive to the respondents behavior depending on the data collect mode.

Like Report

15 years ago

I find it a little offensive to see respondents as not telling the truth and think we do ourselves a disservice by perpetuating this view. The problem is actually the artificiality of the research process and the failure to recognise how people think and act in real life - or how they would think and act if branding and advertising didn't distort their normal behaviour. In the examples given, prospective buyers of MP3 players (and many other products) may well feel, in the early stages of their decision making and while learning about and weighing up alternatives, that model a is superior, no hang on model b's got some other useful feature and model c has some other advantage, then when they finally commit to a decision that model d is the one they want. That's how things happen in real life and is what the group discussion process should ideally replicate. It shouldn't be seen as a problem. Decisions are often not made in a linear or logical fashion and the buying process often involves changes of mind. It's our job to oberve this, not to judge it. In the second example, the notion that allying oneself to an aspirational brand indicates bias is surely missing the point - the whole purpose of aspirational brands is to make people aspire to them. If someone who shops at Tesco says 'I'm Jaguar', then all that tells you is that Jaguar's advertising works. We (the marketing/advertising/research industry) have created a situation in which we have synthesised demand, aspirations and an excess of choice. We expose people to product offerings and communications whose very purpose is to tweak their emotions and undermine rational thought, to create insecurities. In other words, to create stress. So isn't it then a bit rude to patronise people for not being rational or predictable? Especially when they're trying to be helpful by taking part in research.

Like Report

15 years ago

A very interesting and thought provoking article. I would tend to agree with Gill that we need to observe and not judge. This reminds of an incident recently after pre testing of a creative, agency had a question as to why respondents were beahving irrationally (I am in China and people are supposed to behave rationally) and not getting persuaded by the key benefit. We need to learn from the answers that we get from the consumers.

Like Report

14 years ago

i am very much fond of doing research in international relations. i have done my research in hunger and population of pakistan in 2007. now i am doing ms in internatinal relations

Like Report