Net Promoter Score under attack

Serious doubts are being raised about the word-of-mouth metric that claims to predict growth. Robert Bain reports

Tim Keiningham says he probably wouldn’t have a job with ‘loyalty’ in the title if it wasn’t for Fred Reichheld: the man who put loyalty on the map. This is just one of the reasons why debunking Reichheld’s Net Promoter Score is proving an uncomfortable experience for him.

The debate on NPS has been rumbling ever since Reichheld claimed in a 2003 Harvard Business Review article that the simple measure of consumer recommendations was an accurate predictor of revenue growth. It has continued in the wake of his bestselling book The Ultimate Question.

But Reichheld’s claim to have come up with ‘the single most reliable indicator of a company’s ability to grow’ is – according to Keiningham and a team of experts who set out to replicate Reichheld’s methodology – nonsense, based on bad research.

Firms including Microsoft, eBay, General Electric, T-Mobile and Philips will be listening carefully – they have all embraced NPS and used it to predict growth and measure performance.

As he carefully picks NPS apart, Keiningham, VP of loyalty at Ipsos, frequently interjects to underline his admiration for Reichheld and stress the independence of the new study. “I have no axe to grind,” he says. “I’m totally neutral on what the outcome is. If it works, great, it benefits everyone.” But for Keiningham there’s no escaping it: it doesn’t work.

The new study says NPS is no better at predicting growth than other metrics, and that the data backing it up is flawed.

And it’s not just Reichheld’s extravagant boasts about the predictive power of NPS that Keiningham repudiates, it’s also his criticism of the American Customer Satisfaction Index (ACSI), developed by rival consulting firm CFI.

Reichheld said he found absolutely no correlation between ACSI and revenue growth – a claim Keiningham rejects as implausible. “I can look at when geese fly south and have some correlation to how well my football team is going to do this year,” he says. “It may not be significant but it will never be zero.”

CFI founder Claes Fornell, who created ACSI, brushes off Reichheld’s attack on its usefulness as “a minor thing”. He told Research: “You can only take this seriously if you have no knowledge of statistics and measurements. It’s like trying to build a bridge without knowing any structural engineering.” Fornell said that despite the list of high profile proponents of Net Promoter, there was “no evidence that there are many companies that really take it seriously”. “There are big companies whose management have listened to Bain, but they’ll stop using it when they see it doesn’t work,” he said.

Attempts by Research to reach Reichheld for comment while producing this article – and a previous article in March – led nowhere. A glance at his blog suggests the difficult questions may be getting to him. In June last year he wrote: “The reason that so many researchers hate NPS is that so many senior line executives love it.” He continued to defend NPS by saying that while it was less accurate for predicting individual customer behaviour than other measures, it was better at predicting business growth. But a few weeks later he wrote that predicting individual behaviour was the basis of NPS – rather than a correlation to growth.

These recent responses to criticism are characterised by caution, caveats, and more than a bit of confusion – a long way from the grandstanding that initially accompanied NPS.

Meanwhile the non-believers are becoming more vocal. Mark Molenaar of TNS Research Surveys picked up two awards at this year’s South African Market Research Association conference, with a paper arguing that the score is too simple, too narrow and no better than other measures of satisfaction or advocacy.

Keiningham says the response to his deconstruction of NPS has been mixed. While those who never staked anything on the metric can shrug and say they knew it all along, those who have embedded it in their organisations are either in denial, or asking, ‘What now?’

“If this was in the medical field we’d all be up in arms,” says Keiningham. “But we seem to think that’s just the nature of business – no it’s not! The same ethical standards in research have to apply.”

The businesses who have judged themselves on the basis of NPS have the most to lose, but Keiningham fears they won’t be the only ones. Researchers who have ignored their doubts about it in order to please clients will also be hurt, he says.

Furthermore, distrust and disillusionment with NPS have soured the atmosphere in the loyalty field, Keiningham says, and lessons need to be learned.

In a recent seminar on the subject, Randy Brandt, VP of loyalty at Maritz, said the boardroom appeal of Reichheld’s ‘single number’ approach is easy to see. “I think a lot of the blame does fall on us researchers – we do good research but I’m not sure that we always hear what management really wants or needs. So it was somebody who did a good job of that who got management’s ear,” he said.

Keiningham said: “The one thing that the Net Promoter movement has shown is that we need to be better about giving answers to executives that they can embrace and espouse and get people rallying around. But they need to be sound, because once people are burned they’re twice shy, and that makes it harder for us all when a more correct answer comes out.”

July | 2007