Vetting Net Promoter

Last we checked in with Tim Keiningham, SVP at IPSOS Loyalty and co-author of LOYALTY MYTHS, was in December of 2005. At that time I posted some riffs on his LOYALTY MYTHS book.

Keiningham’s book debunked over 50 commonly accepted loyalty marketing practices, one of which was Fred Reichheld’s NET PROMOTER score. In discrediting Reichheld’s NET PROMOTER score, I felt his reasoning was more argumentative then constructive.

Tim emailed me today sharing his recently published article in the Journal of Marketing vetting the Net Promoter measurement. This time around, the reasoning from Tim and his co-authors is much more constructive than argumentative.

---

For some of you we need to backtrack with a quick backstory …

Fredrick Reichheld’s Net Promoter measurement contends companies no longer need to rely on expensive studies and complex statistical models to measure customer loyalty in hopes of increasing sales. Instead, a company only has to ask its customers one question: “How likely is it that you would recommend [company x] to a friend or a colleague?" Knowing the answer to this one question allows a company to easily interpret where it stands in creating net promoters (evangelical customers) which in turn lead to sustainable, profitable growth.

Reichheld so strongly believes in the Net Promoter measurement that he says it is “the single most reliable indicator of company’s ability to grow.” His contention is backed by research done on select companies showing a strong correlation between a company’s growth rate and the percentage of its net promoters. According to Reichheld, “The more ‘promoters’ your company has, the bigger its growth.”

---

With this paper, Tim and his colleagues set out to bring reasonable doubt to the claim by Reichheld that the Net Promoter score is the “single most reliable indicator of a company’s ability to grow.” To scientifically and statistically debunk the Net Promoter score, the authors attempted to replicate Reichheld’s findings using similar data and similar methodology.

I’ll pass on boring you with all the wonky research methodology used because you can read about it in the paper. Instead, here’s the gist of the paper’s findings …

“We find no support for the claim that Net Promoter is the ‘single most reliable indicator of a company’s ability to grow.’

The clear implication is that managers have adopted the Net Promoter metric for tracking growth on the basis of the belief that solid science underpins the findings and that it is superior to other metrics. However, our research suggests that such presumptions are erroneous. The consequences are the potential misallocation of resources as a function of erroneous strategies guided by Net Promoter on firm performance, company value, and shareholder wealth.”

So statistically speaking, the Net Promoter score may not be the “single most reliable indicator of a company’s ability to grow.” Okay … got it … after all, that’s a HUGE claim to make.

However, the Net Promoter score is still ONE indicator of a company’s ability to grow.

And as a marketer, I feel it’s safe to assume the more evangelical customers a business has talking up its products and services to others … the greater likelihood the business is growing and not declining. If all I have to ask, as a marketer, to determine this likelihood of growth, is to ask customers how likely is it that they would recommend this business to others ... SIGN ME UP.

That’s just my take. Read Keiningham’s research paper and decide on your own.

---

UPDATE: Tim Keiningham provided more insights in the comments section.

Comments

[Timothy Keiningham]

John --

Thank you for writing about our Journal of Marketing paper on Net Promoter. I just wanted to provide you with my insights regarding the paper, which differ from the conclusions you have drawn.

1) In terms of the research, we didn't find Net Promoter to be a good predictor of growth at all.

2) Most troubling, Reichheld states that Net Promoter is superior to the ACSI. He reports that their research found the ACSI to have a 0.00 correlation to growth, whereas Net Promoter linked strongly to growth. Besides that being a ridiculous claim about the ACSI (as there are numerous scientific studies showing otherwise), our findings clearly show that when using Reichheld's own data for his best case scenarios, Net Promoter wasn't superior to the ACSI. It is difficult to imagine a scenario other than research bias as the cause of this finding. This is a very big deal. No one accepts bias in medical, physics, or psychological research ... and we don't accept it in the management sciences either.

While you believe that my co-authors were argumentative in Loyalty Myths, Reichheld seems to get a pass with Net Promoter users for his outrageous attacks on other satisfaction and loyalty measures as pseudoscience. He particularly singles out the ACSI, which happens to be a primary source of competitive advantage for a competitor to Satmetrix and Bain & Company: CFI Group, an international research/consulting firm. Therefore, this presents another problem with regard to the claims regarding the research into Net Promoter reported by Reichheld and Satmetrix.

I realize that many have adopted Net Promoter and want to believe; it is fine with me if they even continue to believe regardless of evidence to the contrary. But even those who choose to believe in Net Promoter must demand that the published research used to support their cause be unbiased.

Best...

-- Tim Keiningham

[Bill Gammell]

Ever since I heard of NPS, I have been very interested. I have blogged about one reason I think it is getting the attention that it is: it removes the “friction” that is present in so many metrics. NPS is easy to understand and to spread (sort of a lesson in and of itself about word of mouth).

Anyway, I would agree with you that it should be one tool in the marketer’s toolbox. I only wish that Reichheld had gone into more in depth about an important follow-up question that should always be asked: “why would you (or would not) recommend…’ This is the actionable data that a marketer can use to improve the product and service.

If you’d like, you can read more at - http://ubereye.wordpress.com/2007/01/02/frictionless-formulas (be sure to read the insightful comments from Ron Shevlin – he disagrees with us).

Thanks for your take on NPS.

[Amy Madsen]

Thanks for a neat post. For anyone interested in the benefits companies are receiving from deploying Net Promoter, we have two exciting resources - the Net Promoter Conference blogs. You can scan the "Categories" at right to read about practitioners such as Philips, LEGO, Schwab, GE, Experion, IBM, and more.

Net Promoter Conference: New York
http://netpromoter.typepad.com/npc2007/

Net Promoter Conference: London
http://netpromoter.typepad.com/npc_london_2007/

Other examples can be found in the forum thread "Companies Using Net Promoter." The most recent post references Progressive's use of Net Promoter at their Shareholders meeting:

http://netpromoter.groupee.net/eve/forums/a/tpc/f/2731073251/m/6661046912

[{name withheld}]

The comment from someone apparently from NetPromoter.com seems surreal. Research vetted in the most prestigious journal in the field not only finds that Net Promoter does not link to growth as claimed, but also finds clear evidence of bias in the research reported by Reichheld to support Net Promoter. For this particular case of bias not to be a deliberate attempt to deceive, we would have to assume an incredible level of incompetence. Instead, the issue is not addressed at all: only testimonials. The worst management fads always have testimonials from well known companies (reengineering, CVA, etc.); that's how these bad ideas became management fads in the first place. Instead, I want to know NetPromoter.com's position on the real issue: biased research. I would love to know if it is incompetence or lying.

[Deborah]

I have read many of the blog postings & media interviews regarding the claims that the Net Promoter research was biased and subsequently read in detail the Journal of Marketing paper. I would like to add the following to the conversation.

1. It appears that the Journal of Marketing research referred to shows that Net Promoter is, at a minimum, equivalent to ACSI in correlating to growth. (Figure 1). That says that a simple metric driven by a single question is at least as accurate at predicting growth than a more complex algorithm driven by multiple questions.
2. After discussing the claims with Satmetrix, they stand behind the research. Claiming a research bias is a pretty serious allegation. Both Fred and Satmetrix have spent over 10 years studying the correlation between loyalty and growth and helping companies implement programs that improve loyalty and drive growth. The result of this effort is apparent in reviewing the case studies presented at the Net Promoter conferences: http://netpromoter.typepad.com/npc_london_2007/
3. This conversation is missing the point that is attracting business leaders. The value of Net Promoter is its simplicity. Unlike complex satisfaction indexes, Net Promoter is easy to understand and take action on. Simplified surveys drive higher response rates, a better reflection of the customers that matter, rather than random sampling. Using real-time reporting, leaders can get information in the hands of employees that can address detractors, move the passives and nurture the promoters.

At the end of the day, it’s what customers do to improve loyalty that drives growth. Net Promoter offers an approach that is understandable by everyone in the business, not just the statisticians. This gives an organization a rallying cry for building customer centric organizations.

Full Disclosure: During this conversation I was contemplating joining the Satmetrix team. Having experience deploying Net Promoter programs, and researching the market further, I decided to join Satmetrix as CMO: http://www.satmetrix.com/news/pressrelease_2007-07-17.htm

Let's continue the conversation.

[Tim Keiningham]

Ms. Eastman’s statement that research bias is a serious allegation is absolutely correct. So we are waiting for a good answer. The argument that Net Promoter is “at least” as good as the ASCI is irrelevant. Reichheld claims his research reveals no relationship between the ACSI and growth.

In the Harvard Business Review article that introduced Net Promoter, in the book, The Ultimate Question, and in presentations regarding Net Promoter, the American Customer Satisfaction Index has been specifically mentioned as not linking to firm growth by Reichheld. The book, The Ultimate Question, argues that the ACSI does not yield much insight into loyalty or growth, noting that “investors rarely waste money on standard satisfaction surveys” as a result (The Ultimate Question, p. 86).

Similarly, an article in the Harvard Business Review states (p. 49): “Our research indicates that satisfaction lacks a consistently demonstrable connection to actual customer behavior and growth. This finding is borne out by the short shrift that investors give to such reports as the American Customer Satisfaction Index. The ACSI, published quarterly in the Wall Street Journal, reflects customer satisfaction ratings of some 200 U.S. companies. In general, it is difficult to discern a strong correlation between high customer satisfaction scores and outstanding sales growth.”

Furthermore, in a web-based presentation, Mr. Reichheld states that a “Bain team looked at the correlation between growth and customer satisfaction, and found there is none.” A scatter diagram was shown with the X-axis labeled “American consumer satisfaction index annual growth” and the Y-axis labeled “Sales annual growth.” The R-square reported was 0.00, indicating no correlation whatsoever.

Given that our findings show that Net Promoter was not superior to the ACSI when using Reichheld’s best-case scenarios, it is virtually impossible to imagine a scenario other than research bias as the cause. This is a VERY SERIOUS problem. We expect research published in our most prestigious journals to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science.

Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld’s book, The Ultimate Question. Additionally, biased “research” that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.

It is vital that we not be apologists or revisionists when it comes to issues of research bias. Our credibility must never be in question regarding the research we publish in prestigious journals; the truth matters. Therefore, discussions about Net Promoter by researchers (practitioner and academic) must first adequately address this issue. If not, then why do any research at all, as we can simply present the answers we want to believe as supporting evidence and be done with it? Given the evidence we uncovered, however, we seriously doubt that there will be an acceptable answer to the issue of bias in Reichheld's reported research. [Ironically, in The Ultimate Question, Reichheld emphasizes the importance of eliminating bias from research (pp. 106-111).]

While we can all agree for the need to have measures that are easily understood and used by managers, that is completely irrelevant to the issue at hand. Regardless of whether or not one chooses to believe in Net Promoter, we all must insist that the evidence used to support the metric be unbiased. This issue is so overridingly important to very core of what we as researchers (academic and practitioner) do and stand for that it must be addressed. Our credibility in science and in practice is based upon honest and fair evaluations of data...if this is contaminated and unchallenged, then there is no reason to believe anything we say.

Sincerely,

Tim Keiningham