Truth, lies and market research

One of the key priorities, if not duties, of our trade is to design research in a way that elicits honest answers from people. But even with sound design and the best of intentions on the part of researchers and respondents, market research can fail.

Getting a thumbs-up from research but a thumbs-down in the marketplace is a nightmare for any project. It might happen as a result of sampling or interview error, or because respondents are bored or taking part for the wrong reasons. But it might also have to do with psychological processes in participants’ minds that cause them to respond in a way that misrepresents their true opinions and behaviour.

The psychological processes that underpin respondent error can broadly be split into group and individual processes. Group processes are more relevant to qualitative research methodologies where participants are exposed to others who could shape their behaviour. Individual processes, however, could crop up in any sort of research and both, to varying degrees, can be at play simultaneously during the same project.

The power of the herd
Let’s take the group processes first. Focus groups are a perfect example. Very often decisions made following focus groups are worth millions, and they present possibly the biggest minefield for respondent bias.

Imagine you are launching a new product, an MP3 player for example, and you are running groups to select the best one. Throughout the session your skilled moderator has got participants to discuss the pros and cons of six models, and the group settles on two preferred options. But when at the end of the session the moderator invites participants to take one of the specimens home, seven out of eight of them opt for one of the players that didn’t make the shortlist.

These things do happen, and social psychology offers numerous potential ways to explain why:

• Herding – people unwittingly follow others. This could mean that, irrespective of what the product might have been like, the first two people may have picked the same thing, thereby prompting a chain reaction that overtakes the capacity for individual rational evaluation.
• Normative social influence – people behave in a certain way to fit in. This might suggest that participants had not really clicked with any of the product features and that any benefits were insufficient to avoid being overturned by rudimentary group dynamics.
• Informational social influence – people struggle to make independent judgements and instead look to others for validation. In this case, they may not have fully comprehended the benefits of the propositions or differences between them and renounced their own opinion in favour of others’.

It’s important to take two things from the above scenario: participants’ responses can be affected by a range of factors during the research process, and people might be pretty bad at predicting their own behaviour (or that of others), even if they are convinced of their ability to do so.

And it can get even more complicated. Research in the 1990s showed that any group processes are amplified proportionally to the perceived importance of the task at hand. So in our scenario, participants’ awareness that someone would be using their input to make decisions about a product launch could have affected their desire to give the right answer rather than express their honest opinion.

Lonely souls
Individual research participants can be equally vulnerable to bias. Inevitably, part of this has to do with the back-end of research design. For instance, the ‘inability error’ – which causes respondents not to answer a question or answer it untruthfully, often has to do with inappropriate sampling or question design. But the fact that many respondents choose to answer a question they don’t know the answer to reveals a more internal biasing process often referred to as the ‘best light’ phenomenon.

Imagine a survey asking respondents which brands best represent them. The truthful answers might be things like Tesco, the Daily Mail and Vauxhall. But many people will opt for more aspirational brands instead: the Rolexes, Jaguars and Armanis of this world become standard responses even if they have precious little connection with the respondents’ day-to-day lives. The interviewee attempts to project a positive image of themselves and in doing so biases their answers, perhaps because they are trying to make the interviewer like them, or give their self-esteem a lift.

Truth serum
Good design, testing and execution need to be observed at all times to avoid these pitfalls. There are also a few further steps you can take. In qualitative research, it is advisable to:

• consider triangulating methodologies. Observation studies or ethnographic approaches can go a long way towards neutralising respondent bias;
• make the most of non-verbal communication. A good qualitative researcher is able to pick up non-verbal cues from research participants and cross-examine those against verbal responses;
• engage participants before and after the group. This can be done online using tools that allow participants extra familiarisation and expression, and enable the researcher to gain additional insight into group members and their contributions.

There are two particular recent examples of successful incorporation of psychological thought into quantitative market research. BrainJuicer has put together an online concept testing methodology called Predictive Markets, built on Galton and Surowiecki’s crowd behaviour studies. By aggregating individual responses to tested concepts, they sidestep the challenge of ‘groupthink’, whereby physical groups end up supporting bad ideas. This approach is a compelling marriage of the benefits of crowd opinions with the ability to eliminate the shortcomings that stem from group processes.

Another technique called Implicit Attitude Testing (IAT) is widely used by a working group at Harvard University. It was recently applied in the UK to a project looking at attitudes to facial disfigurement. Rather than asking participants to self-rate their own prejudice, this online approach tests the associations between words and images that people make and measures the time taken to respond to different pairings of stimuli. By focusing on subconscious rather than conscious psychological processes, IAT is capable of eliciting deep-laid attitudes to social phenomena that are very susceptible to normative or ‘best light’ respondent bias.

These examples show how an understanding of the psychology behind why respondents give the answers they do can be an integral part of the research process, rather than something that is just bolted on. Market research isn’t a discipline that exists in isolation. Good research should bring together a wide range of skills and disciplines, and this one should be a key consideration for agencies if clients are to be sure the decisions they make are the right ones.