Net Promoter System Imperfections

Net Promoter System Imperfections

Over the last decade, Net Promoter Score® has gradually become the new standard for measuring customer loyalty and satisfaction, not just for startups but for businesses around the world.

Popularly labeled as the ‘ultimate question’, NPS® survey has made the long customer satisfaction surveys obsolete (for good!), and introduced a new, simple and convenient way for businesses to measure customer happiness.

But despite its widespread adoption and incredible simplicity, experts around the world have argued that the system has inherent flaws. From not providing actionable metrics to having high statistical variance, NPS has been criticized for painting an inaccurate picture of customer satisfaction.

In this article, we explore some of the biggest criticisms of Net Promoter Score, and understand why it has failed to deliver favorable outcomes for a few brands.

It’s a survey of feelings

NPS cannot be considered a quantifiable metric, since it calculates how many users are “likely to refer your brand”, not how many users “actually referred your brand”. It’s a leading indicator of your brand’s referral propensity, measuring the “inclination” of customers to refer (feeling), instead of telling you how many people did refer (analytics).

For instance, it’s entirely possible that you have an NPS score of 70, but your brand promoters are not sending enough referrals, while your competitor that has an NPS score of 40 is driving more referrals, sales, and traffic.

Of course, if you had to create a framework that counts the number of referrals that every customer actually makes and compare that with how they feel about your brand, it could turn out to be a much more complex implementation than NPS.

But since there’s no way to actually tie up the number of referrals a customer makes with the number of referrals they intended to make, it’s safe to say that NPS captures emotional responses and can, therefore, be termed as a “survey of feelings”.

You cannot slice and dice the data

NPS is a simple metric that lets you measure customer happiness and brand loyalty. But that simplicity can turn out to be a serious limitation for businesses who have no clear way of interpreting the number.

For example, let’s say your business had an NPS score of 50 previous year, which increased to 55 recently. Now, from a statistical viewpoint, it looks like you have a net 10% YoY increase, which is a great thing. But, it gives you no insights on why it happened.

For example, if your website visits went from a thousand to a million in a couple of weeks, you could drill down on the location, type of traffic, the specific pages users are visiting, the search terms that users are searching, and understand what’s driving the trend.

What can you make out from the fact that your NPS increased 10% YoY? Can it be co-related with your net increase in sales or profits? If yes, on what basis would you establish that relationship? If NPS measures customer happiness, why are customers happier this year? The number alone provides absolutely no answers.

To address this limitation, many businesses now ask a follow-up question after they capture the NPS feedback score, something like, “What is the most important reason for your score?”

Some NPS services offer segmentation to sort the score for a particular customer group of the business. Typically you’ll be able to segment them by plan type, account creation date, location, etc. This approach can give you the specific evidence of what segment drives your score down and where the key drivers of your business are.

It has weak audience orientation

If you’re running a B2B business, capturing responses for NPS surveys can be a little tricky, since there’s no clear way of identifying the client’s actual perception. Different departments can engage with you at the same time, and have entirely different opinions about your product/service.

For example, a junior executive in the customer service department might give your product a 9 on the survey, but a senior management executive can give you a 5 or 6. Whose opinion matters more to you?

Without appropriate segmentation, NPS surveys can inaccurately aggregate user opinions, thereby questioning the credibility of your NPS.

As we mentioned in the previous paragraph, to solve this problem you can segment your customers based on the job title. Isolate the NPS for each department to find out the level of their satisfaction and learn what obstacles they faced throughout the process.

With these valuable insights on hand, you would be able to prioritize your challenge areas to improve customer experience for those departments.

You need to be mindful of the geographical areas

Most businesses miss out on this one.

Apart from segmenting opinions based on individual departments, you also need to be mindful of the regional differences, as taking NPS score on an absolute basis can paint a very wrong picture of the market.

For example, European customers are generally much more passive than American customers, even though both of them really like the product.

So, you can have an NPS score of 60 in America and only 45 in Europe, but that still does not imply that you’re doing bad in Europe. It just means that Americans and Europeans don’t express their referral propensity the same way.

Since NPS captures how people feel and not how they act, you need to consider the demographics of the region and adjust the NPS score for variations, before deriving any conclusions out of it.

Get your survey timing right

Although NPS has gained popularity for being ‘the ultimate question’ that replaces long customer satisfaction surveys, it still has to be asked in the right context, in order to capture unbiased user opinions.

Depending on how it’s implemented, NPS popups can either increase engagement or annoy users. For instance, you don’t want to ask users who have just onboarded your product, whether they would recommend your product (they just don’t know it yet!). But, as you go further down the funnel, you might want to figure out the best moments to ask the question.

For example, if you’re going to ask your customers whether they would recommend your product, while they’re waiting for an output (loading/processing screen) or just after they had a delightful experience (order completion page), there are chances that you would capture honest responses. But if you’re going to pop that question while they’re in midst of an important workflow, they’re going to close the pop-up and forget about the survey.

In case you decide to use a non-obtrusive pop-up or text banner, only those customers who have used your product long enough would be able to notice the difference. For majority of the users, the change would be unnoticeable, thereby skewing your result set.

Considering how timely NPS pop-ups need to be in order to capture honest emotional responses, some businesses use emails to send NPS questionnaires, so that users can fill it in at their own convenience.

The downside of using email for sending NPS surveys is its low response rate, which is around 20-30%%. But there are NPS services that offer advanced features that can help you increase the response rate.

One of them is sending the survey in your customer timezone, as well as resurveying – resending your NPS survey to those customers who didn’t open it.

But even emails can capture varied responses since NPS is a survey of feelings. It’s possible that instead of reflecting back on the actual experiences, people are driven by the mood they’re in (happy, sad or angry), when they hit the reply button.

Also, if you’re only using emails to conduct NPS surveys, you’re limiting your audience to users whose email you have or who have opted in for your emails and open them (brand promoters), and not capturing responses from audience that are not engaging with your product (brand detractors) or don’t open your emails (passives).

Perhaps, the best way to get past this emotional barrier is to identify channels that have a higher response rate within your industry and focus on interactions that have less criticality and high interactivity.

There’s no uniform NPS Benchmark

NPS does not have a uniform benchmark. For instance, if your growth rate is 20% or your SEO page rank is 3, you know where you stand with respect to other companies in the world. But can the same be said about NPS?

According to a study, average NPS score for investment firm lies in the range of 55-65 with the lowest being 30, while average NPS score for internet service providers lies in the range of 5-10, with the lowest being -30 and highest being +30.

Therefore, you cannot say a lot about a company just by looking at their absolute NPS score, without considering their relative performance within the industry. It’s imperative to close the loop and understand why your customers gave you the feedback.

For instance, in the above case study, customers who go to department stores have less critical and highly interactive shopping experiences. If the department store next door is closed for the weekend and you can’t get your groceries, you’re not going to get angry.

But what if your Internet connection was down for the weekend due to maintenance issues? Since ISPs are delivering highly critical, highly interactive experiences, even a small outage of a few hours can piss off the customers.

That being said, even within the same industry or product vertical, NPS is problematic to interpret, since there is more than one way to get the same number and there are different factors that may affect the NPS benchmarks.

For example, let’s assume that two businesses in the same vertical have an NPS score of 20. Would you consider both of these companies to have delivered the same level of customer happiness, especially when the first business could have 20 promoters, 80 passives, and 0 detractors, while the other could have 60 promoters, 0 passives, and 40 detractors? Would it be fair to equate a business that has 40% detractors with the one that has 0% detractors?

I don’t think so.

It’s statistically challenging to be consistent

While there’s a great advantage to measuring your NPS on a periodic basis, getting an accurate NPS can be statistically challenging as you need a lot of responses to get your variance down to a minimum level.

Mathematically speaking, NPS equals the percentage of Promoters (9s and 10s) minus the percentage of Detractors (0s to 6s). Now, since we’re subtracting two values with high statistical variance and eliminating the middle-pack Passives (7s and 8s) altogether from the calculation, the individual readings can become spread out, leading to inaccuracy in calculating the NPS score.

Most brands make the mistake of either surveying all their customers only once or twice a year and don’t expand their sample size steadily to reach every customer. The result? They capture feedback with a high degree of response bias, which leads to an irrelevant NPS score.

To ensure that you have a statistically significant NPS survey, you need to schedule NPS surveys throughout the year and work towards increasing your response rate.

It can be done by starting with a small subset and then systematically surveying all your customers. Emails are an excellent choice to increase coverage and consistency, as they’re non-intrusive and can be auto-scheduled to maximize user participation.

Most companies don’t know what to do with it

Ever since, Fred Reichheld, a partner at Bain & Company, created a new and simple way of measuring how well a business treats its customers, it has evolved from being a loyalty metric to a completely new management system that puts the customer at the epicenter of organizational goals.

But for most organizations, when looking to leverage the power of NPS to reduce churn and customer dissatisfaction, the lack of a customer success framework can be a real issue. Most companies simply don’t know what to do with the data.

For example, if the sales are down by 20%, businesses usually know how to formulate an execution plan. But do they have an actionable plan for NPS going down by 20%?

As most businesses can’t formulate actionable strategies and don’t have a framework to accommodate the results, NPS surveys become largely ineffective in driving organizational change and improving the impending outcomes.

Whereas what they need to be focusing on is closing the customer feedback loop. The goal is simple – delight detractors by fixing the problems they highlighted, surprise passives to win their loyalty and engage promoters by giving them the means and instruments to do it.

Closing customer feedback loop
Closing customer feedback loop

Does it still make sense to know Your NPS score?

Despite the above-discussed limitations, it still makes sense to integrate NPS surveys in your feedback loop, for the simple reason that not having any feedback is not always a good sign.

If you’re not capturing how users feel about your brand, you’re not measuring how happy your customers really are with your product or services.

Considering the fact that 90% of buying decisions are driven through emotions, and not rationality, capturing how users feel about your brand is vital for long-term sustainable growth. While NPS certainly has its own limitations, most of them can be overcome through proactive implementation and predetermination of actionable metrics.

Retently makes it easy to measure the NPS score of your brand. With quick installation and easy-to-follow documentation, you can now get actionable insights on how to reduce customer churn, grow revenue, and build long-term customer happiness. Sign up for a free trial to start measuring your customer satisfaction.

Get notified of new articles Leave your email to get our monthly newsletter.