FEATURE1 July 2007

Net Promoter Score under attack

Serious doubts are being raised about the word-of-mouth metric that claims to predict growth. Robert Bain reports

Tim Keiningham says he probably wouldn’t have a job with ‘loyalty’ in the title if it wasn’t for Fred Reichheld: the man who put loyalty on the map. This is just one of the reasons why debunking Reichheld’s Net Promoter Score is proving an uncomfortable experience for him.

The debate on NPS has been rumbling ever since Reichheld claimed in a 2003 Harvard Business Review article that the simple measure of consumer recommendations was an accurate predictor of revenue growth. It has continued in the wake of his bestselling book The Ultimate Question.

But Reichheld’s claim to have come up with ‘the single most reliable indicator of a company’s ability to grow’ is – according to Keiningham and a team of experts who set out to replicate Reichheld’s methodology – nonsense, based on bad research.

Firms including Microsoft, eBay, General Electric, T-Mobile and Philips will be listening carefully – they have all embraced NPS and used it to predict growth and measure performance.

As he carefully picks NPS apart, Keiningham, VP of loyalty at Ipsos, frequently interjects to underline his admiration for Reichheld and stress the independence of the new study. “I have no axe to grind,” he says. “I’m totally neutral on what the outcome is. If it works, great, it benefits everyone.” But for Keiningham there’s no escaping it: it doesn’t work.

The new study says NPS is no better at predicting growth than other metrics, and that the data backing it up is flawed.

And it’s not just Reichheld’s extravagant boasts about the predictive power of NPS that Keiningham repudiates, it’s also his criticism of the American Customer Satisfaction Index (ACSI), developed by rival consulting firm CFI.

Reichheld said he found absolutely no correlation between ACSI and revenue growth – a claim Keiningham rejects as implausible. “I can look at when geese fly south and have some correlation to how well my football team is going to do this year,” he says. “It may not be significant but it will never be zero.”

CFI founder Claes Fornell, who created ACSI, brushes off Reichheld’s attack on its usefulness as “a minor thing”. He told Research: “You can only take this seriously if you have no knowledge of statistics and measurements. It’s like trying to build a bridge without knowing any structural engineering.” Fornell said that despite the list of high profile proponents of Net Promoter, there was “no evidence that there are many companies that really take it seriously”. “There are big companies whose management have listened to Bain, but they’ll stop using it when they see it doesn’t work,” he said.

Attempts by Research to reach Reichheld for comment while producing this article – and a previous article in March – led nowhere. A glance at his blog suggests the difficult questions may be getting to him. In June last year he wrote: “The reason that so many researchers hate NPS is that so many senior line executives love it.” He continued to defend NPS by saying that while it was less accurate for predicting individual customer behaviour than other measures, it was better at predicting business growth. But a few weeks later he wrote that predicting individual behaviour was the basis of NPS – rather than a correlation to growth.

These recent responses to criticism are characterised by caution, caveats, and more than a bit of confusion – a long way from the grandstanding that initially accompanied NPS.

Meanwhile the non-believers are becoming more vocal. Mark Molenaar of TNS Research Surveys picked up two awards at this year’s South African Market Research Association conference, with a paper arguing that the score is too simple, too narrow and no better than other measures of satisfaction or advocacy.

Keiningham says the response to his deconstruction of NPS has been mixed. While those who never staked anything on the metric can shrug and say they knew it all along, those who have embedded it in their organisations are either in denial, or asking, ‘What now?’

“If this was in the medical field we’d all be up in arms,” says Keiningham. “But we seem to think that’s just the nature of business – no it’s not! The same ethical standards in research have to apply.”

The businesses who have judged themselves on the basis of NPS have the most to lose, but Keiningham fears they won’t be the only ones. Researchers who have ignored their doubts about it in order to please clients will also be hurt, he says.

Furthermore, distrust and disillusionment with NPS have soured the atmosphere in the loyalty field, Keiningham says, and lessons need to be learned.

In a recent seminar on the subject, Randy Brandt, VP of loyalty at Maritz, said the boardroom appeal of Reichheld’s ‘single number’ approach is easy to see. “I think a lot of the blame does fall on us researchers – we do good research but I’m not sure that we always hear what management really wants or needs. So it was somebody who did a good job of that who got management’s ear,” he said.

Keiningham said: “The one thing that the Net Promoter movement has shown is that we need to be better about giving answers to executives that they can embrace and espouse and get people rallying around. But they need to be sound, because once people are burned they’re twice shy, and that makes it harder for us all when a more correct answer comes out.”

July | 2007


14 years ago

I think the ramifications to middle and lower management should also be discussed. There is an increasing trend to now tie financial incentives and compensation in with NPS. And with a metric that's so higly suseptable and easily manipulated its created a dirty little secret in these companies. It's just to easy to munipulate. Tell only those who would give you a high score to take a survey. Tell people to give you a ten and refrain from letting detractors voice their concerns. Hold receipts back from potential detractors...on and on.

Like Report

14 years ago | 1 like

I find it interesting that the most negative comment about the NPS metric is that it is "no better" than any other metric currently in the marketplace. However the commentary that follows seems to completely dismiss NPS as a valid metric. Let's be honest here. Worse case scenario, according to these studies, NPS is at the very least in the same league as any other metric currently available in the marketplace, but typically at a fraction of the cost to implement. I guess the proof is in the pudding! I have yet to see or read about a significant number of companies abandoning the NPS metric. Its very easy for Ivory Tower types to dismiss the NPS metric. Now how about putting your money where your mouth is. Find an alternative that is: as cost effective, simple for everyone to understand and at the very least as accurate as any other metric available. Then we can talk. As for the dirty little secret posted by Mr or Miss Anonymous, gaming a customer satisfaction survey can be associated to any of the metrics. It is an execution issue and not an overall reliability issue of any of the metrics. Poor execution is poor execution no matter what the metric is. Case in point, many (if not most) car dealers offer bonuses to customers for a perfect score in their surveys, and few use the NPS metric. Seems NPS has some in the professional market research industry shaking in their boots. In my experience, those that fear something, are the ones that have something to hide and typically the most to lose! PS: Anonymous, when you so recklessly sling mud based on inaccurate information, at least have the courage to identify yourself.

Like Report

14 years ago

There is indeed a bit of tooing and froing in the NPS space. I generally come down on the side of Guy. I use it because it works and is relatively easy to implement. However, if you want a more detailed list of pro and con research have a look at this blog post that compiles the public research for both sides: http://genroe1to1.genroe.com/2010/10/25/net-promoter-score-research-the-for-and-against-list/ .

Like Report

14 years ago

Is it possible for an employee to be held accountable for these results? For some of the big companies utilizing this system, the fear of losing employment because of someone's mear opinion could be a determining factor of whether or not they are doing their job to the best of their ability.

Like Report

11 years ago

@guy & @adam - thanks for your considered response, and link to the great post. I would love your thoughts on a LI discussion thread I started about NPS - my hypothesis is that Recommending a product to a STRANGER is a better measure/predictor of brand loyalty than recommending it to friends/family/colleagues. You can find it here http://ow.ly/qIDA0 ......looking forward to hearing your thoughts.

Like Report

11 years ago

Guy you are wrong, emphatically wrong. If a company bases a metric on stats then they need to be prepared for a debate on the statistical validity of their argument. Not say, oh you are an ivory tower type - that is a very serious error. So, lets look at the stats: NPS is virtually the same as CSAT that is a fact, analysis of responses consistently comes up with correlations above 0.8. The economic arguments plied are also highly suspect - change detractors to promoters - yeah right, lets make the world perfect and everything will be super. Tim is correct, however, the advantages of NPS are cultural. Executives generally are of the judgemental, stentorian type so they get what they deserve. Of course NPS works in identifying things that go wrong, in exactly the same way as disatisfaction would work as a metric (a better metric). So there is your simpler metric. However, you want a magic metric, there isn't one period. Metrics are outputs not inputs, have a great idea about a product or service everyone wants to buy, implement it well and you'll increase your revenue. ALL ELSE iS COMPLETE BS GUARANTEED

Like Report