Ask the right questions...

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

We know from the many discussions around referendum wording that the way in which a question is phrased can have a significant impact on the response generated. 

This is also true for surveys with more than one question, but – in those instances – another factor can influence the outcome: survey framing. Recent research has revealed that beginning a survey with an open-ended question – specifically one asking for positive feedback – can lead to increased customer purchases among those completing the survey. 

Previous research has demonstrated the ‘mere-measurement effect’, by which simply measuring an individual’s purchase intentions changes their subsequent behaviour in the market. This latest study proposes ‘mere-measurement plus’, whereby starting a survey by asking customers to recall something positive about their purchase experience increases subsequent sales. 

The research used two studies to test this hypothesis fully; the first was a longitudinal ( 12-month) field experiment with customers of a large, US, portrait-studio retail chain. After each interaction, all customers were invited to participate in a survey. 

The company’s standard feedback survey was modified so that customers were assigned to one of two groups – in one they were given an open-ended prompt at the start of the survey, asking them what had gone well with their most recent visit; in the other group, customers were asked only closed questions. 

Except for the opening question in the first condition, the questionnaires were identical. 

Even after accounting for a number of effects – including ‘mere measurement’, previous product and service quality, past purchases and income – it was found that customers who had been asked the open-ended question at the start of the survey spent 8.25% more than those who were not. 

Because of constraints with the first study around assessing the baseline effects of solicitation and some aspects of mere-measurement, a second study was carried out with a manufacturer of business to business (B2B) software that continually tracks customer feedback. 

Four customer groups were created: those who weren’t sent a survey; those who were sent a survey but didn’t respond; customers who completed a survey containing only closed questions; and those who completed a survey that started with a request for positive feedback. 

The comparison of the average customer spend across the groups was telling: those who were not sent a survey spent $0.17; those who were sent a survey but chose not to respond spent $1.27; people who completed the survey of closed questions spent $6.66; and customers who completed the survey with the positive-solicitation question spent significantly more on average – $8.85. 

“The typical approach to understanding customer attitudes often assumes that customers’ perceptions are static once experienced – in soliciting feedback, researchers simply try to capture the customer’s image of that experience,” the research authors wrote. “In contrast, our field experiments reveal that customers’ perceptions of their experiences are malleable, and that an open-ended, positive-solicitation frame can encourage future customer spending.”

positive narratives 

The results of the study raise questions around whether market researchers should be wary of using such questions for fear of skewing survey results. 

Sterling Bone, one of the authors, agrees that some care should be taken in using this framing approach, but he still believes that researchers should embrace the approach to “understand the basis of customer satisfaction and customer loyalty”. 

“Capturing the positive narratives in customer experience allows firms to rally their employees, celebrate their successes, and to benchmark customer experience internally,” he says. 

“As with any measurement, researchers need to recognise that these questions trigger behaviours in respondents and they need to be careful in how they are used, from a practical and ethical perspective.

“We caution researchers to understand the true motives of using open-ended, positive-solicitation questions. If the motive is to reframe entire surveys to focus on positive experiences, a company could misinterpret a manipulated uptick as an actual improvement in service.” 

But it’s not just the potential misinterpretation of results that raises issues, according to Dr Michelle Goddard, director of policy and standards at the Market Research Society (MRS). She says there are also questions around the integrity of the approach, and suggests the “bias by design” that this approach appears to advocate “risks undermining the wealth of robust research undertaken by research practitioners”.

“Rules on data collection in the MRS code of conduct seek to ensure that bias in research design and methodology is minimised,” says Dr Goddard. “The code specifically requires that data-collection processes must be fit for purpose and that clients are advised accordingly. It also requires that participants not be led towards a particular point of view.”

Given that structuring research in this way lends itself to potential contravention of the MRS code, Dr Goddard suggests that – as a minimum – any potential client looking at a project of this type would need to be advised that it is not a research project, but more of a marketing exercise. 

relationship building

Bone, meanwhile, recommends that companies continue to offer their original surveys to a control group of customers, so that genuine service trends are not obscured. 

“If the researcher’s motives are to promote positive wellbeing in customers, they can gain comfort from knowing that previous work in psychology points to the salutary effects of expressing gratitude, and suggests that being asked for compliments might increase customers’ feeling of wellbeing,” he says. “We suggest researchers consider this approach less as a means of manipulating perceptions, than as a way of building relationships.” 

Sterling A. Bone, Katherine N. Lemon, Clay M. Voorhees, Katie A. Liljenquist, Paul W. Fombelle, Kristen Bell Detienne, and R. Bruce Money ( 2017 ) ‘Mere Measurement Plus’: How Solicitation of Open-ended Positive Feedback Influences Customer Purchase Behaviour. Journal of Marketing Research: February 2017, Vol 54, No 1, pp156-170.