The Froot Loops NPS Case Study (Part 2) Methodology and Quantitative Results
Goals
Overall I wanted to learn more about Net Promoter Score (NPS), what is good about it, and not good about it.
Specifically I wanted explore the following questions:
- What does the NPS score mean?
- Is it confusing to take a 0 to 10 scale and turn it into a -100 to +100 score?
- What are we missing when we ignore people who give 7/10 and 8/10 responses?
- Is the 0 to 10 scale difficult to rate?
- Is the NPS question awkward and difficult to answer?
- What can a single scale response tell you?
Methodology
On June 8, 2017, I approached attendees of the UXPA 2017 conference in Toronto, Canada one at a time and asked if they would like to try Canadian Birthday Cake Froot Loops (BCFLs). If they agreed I would have them eat some and ask…
“How likely or unlikely would you be to recommend Birthday Cake Froot Loops to a friend or colleague? With 0 being not at all likely and 10 being extremely likely.”
I then recorded their NPS score in a notebook. I also captured qualitative comments.
Participants
I approached 61 attendees and 51 agreed to try the BCFLs. I did not track demographics but I can generally approximate.
There were slightly more female than male participants. Most were in their 30s and 40s, then 20s, and some were 50s and above. Most were English-speaking but some required translating (from French). Most were from the United States, second most were from Canada, and the rest from a variety of countries.
Since it was a collection of UXPA members awareness of NPS was a bit higher than normal (but not as high as I expected). Overall feedback may have been more open and critical than from “civilians.”
Lastly, feedback captured later in the day skewed slightly more positive when more participants were drinking alcoholic beverages. My note taking may have also suffered for similar reasons.
Results
Below you can see the raw scores using the 0–10 scale. The most frequent score was a 0 which was given 11 times.
I counted up the the scores and put them into the Delighted NPS Calculator to get the final NPS score which was -64.
Now I knew the final score: -64. What I did not know is what that means. Since it was a large negative number I assumed it was pretty bad.
I searched for NPS scores to compare against. I found a 2016 report by Temkin Group which collected NPS scores from 315 companies which is charted below. USAA had the highest score (68) and Comcast had the lowest score (-5).
Discussion
What does the NPS score mean?
It does not tell you much on its own. The NPS score has meaning when it is compared with other NPS scores. The BCFL -64 NPS score is way off-the-chart when compared to scores in the Temkin report. In fact it is so much worse that I wonder if I calculated it correctly.
The BCFL NPS score could serve as a baseline to measure future Froot Loop flavors against.
Is it confusing to take a 0 to 10 scale and turn it into a -100 to +100 score?
Absolutely. I originally calculated a mean of the 0–10 responses and thought that was the NPS score. Converting it with the formula seemed overly-complicated and very black box. It also negated any patterns I noticed in the range of responses.
Jeff Sauro makes an excellent point about this conversion in the MeasuringU blog.
“The main disadvantage of the Net Promoter Score is that it reduces an 11 point scale into a 3 point scale.”
Sauro mentions two consequences of this: increased margin of error and differences are harder to detect. You need a larger sample and it is more difficult to see declines or improvements. For example if you have a lot of 6s in one study and have the same number of 0s in another study, that difference is partially erased in the conversion. A 6 and a 0 are both detractors so they get scored the same.
What are we missing when we ignore people who give 7/10 and 8/10 responses?
Five respondents gave BCFLs an 8 which makes it the third most frequent score given. That seems like a significant omission but I cannot identify any meaningful quantitative impact. Even though the 7s and 8s are not part of the final calculation step, they still “count” because we are working with the remaining percentages. They are not ignored at all.
Is the 0 to 10 scale difficult to for respondents to rate?
I could not tell by the NPS scores alone. (more on this in part 3)
Is the NPS question awkward and difficult to answer?
I could not tell by the NPS scores alone. (more on this in part 3)
What can a single scale response tell you?
The broad conclusion from this exercise is that Canadian Birthday Cake Froot Loops were not well received and people would not recommend them.
That is all I learned from NPS but this is not the end of the Canadian Birthday Cake Froot Loops story.
Like any good qualitative researcher I asked “why?” and learned a lot more!
(#FLNPS Part 3: Qual to the Rescue!)
