Net Promoter Score volatility and frustration
Most companies are focused on continuously improving their customer satisfaction, so tracking a business Net Promoter Score is an important step in developing the culture of Customer Success. Over the past years, Net Promoter Score has proven to be a customer satisfaction key metric.
If you, however, are only focused on tracking the score just for the sake of doing it, you are losing time and efforts. The score itself is highly volatile, and the greater shake we see here and the instability it causes is what’s behind the Customer Success department frustration. Having to explain to the management that an increase or decrease in the score that looks considerable is an unpleasant task at best. It generates possible windy discussions and at worst, may undermine the credibility and the entire effort of the Customer Success team.
The science behind Net Promoter Score volatility
While Net Promoter Score is a metric that provides a quantification of your customer’s satisfaction, there is no absolute number of clients that you should interview during your customer satisfaction survey in order to get a statistically significant NPS. That number depends on several factors and differs from business to business. For some B2B companies getting 50 NPS survey responses might be significant, while some B2C companies would need at least a few thousand responses just to consider the results statistically reasonable.
If you only have 50 clients, getting 50 responses would mean 100% and there is no chance you can have a sample of 100, 200 or 500 customers. At the same time, if you have 1,000 clients, getting 50 responses would only mean 5%, so it is hardly the same significance.
There are several factors that need to be taken into account for a statistically significant Net Promoter survey: population size, response rate, margin of error, confidence level and standard of deviation. Consult this article for additional information on how to calculate this.
Let’s look at this example for an average to above-average company: 50% promoters, 30% passives and 20% detractors, with a population size of 5,000 and a response rate of 20%, so the amount of respondents being n=1,000. The NPS score would be 30 (50% – 20%).
The margin of error (MoE) for this score is +/- 2.5, meaning that if we repeated this study the same way 100 times, we’d expect that about 95% of the time we’d end up with an NPS rank varying between 25 and 35.
In order to increase the precision of this NPS campaign twice, we’d actually need a base size of over 3.8 as many respondents!
n * (P1/P2)2 = 1,000 * (4.9/2.5)2 = 1000 * 3.8416 = 3,842
(where P1 and P2 is the Precision factor)
In a typical 14% response ratio scenario we would need an amount of x 27 more customers (3.8 / 0.14 = 27).
This agrees with the statistician’s rule that if we want to cut precision or margin of error in half, we must quadruple our sample size!
Choosing the right segment and addressing the feedback
For business impact, it is also important to consider the quality of the sample, not just the numbers. This covers what customer segment are you surveying, how long have they been your clients, how engaged are they with your product and service, etc.
Closing the feedback loop and addressing the issues reported by customers has a greater impact on a business than getting a statistically significant NPS survey result. Once you start receiving feedback, classify it by priority and start addressing the reported issues.
For this reason, it is recommended to start running Net Promoter Score surveys right away, even if you have just a few customers. In this way, customer satisfaction will be implemented at the core of your company, as an important part of the Customer Success culture.