In our surveys, we use a five-point satisfaction scale, ranging from “Very Satisfied” and “Somewhat Satisfied” over “Neutral” to “Somewhat Dissatisfied” and “Very Dissatisfied”. Now, we could also write a blog (or several blogs) about the nature of satisfaction scales. Suffice it to say that the academic discussion about which satisfaction scale to use probably predates “Hamlet”, and is equally undecided. We prefer the simple and intuitive five-point scale: our respondents have businesses to run, and we need to make it as easy as possible for them to respond. Needless to say, once a survey is running, it is difficult to change scales, as we would lose historical comparability – in our case, more than ten years.
So, based on the five-point scale, what metrics are available? We will take a closer look at four candidates:
- Top Box Score: This is calculated by dividing the number of “Very Satisfieds” by the total number of respondents for a particular question: (Very Satisfieds)/(Total number of respondents). This is the “official” metric for our North American Surveys.
- Top Two Box Score: Similar to “Top Box”, but adds the Second Box (“Somewhat Satisfieds”) to the numerator, so: (Very Satisfieds+Somewhat Satisfieds)/(Total number of respondents). We used this metric until 2012. We will explain why we switched to Top Box in a bit.
- Average Score: For this calculation, we convert the score labels into numeric values: “Very Satisfieds” = “5”, “Somewhat Satisfieds” = “4” … down to “Very Dissatisfieds” = “1”, and then simply calculate the average. Traditionally, we have not used this metric for reporting scores.>/li>
- Net Promoter Score (NPS): To be clear: this methodology is rarely used to measure satisfaction, but the strength of customers’ recommendations and advocacy. NPS typically uses a 10-point scale, where “9” and “10” are the “Promoters”, “7” and “8” the “Passives”, and the rest are “Detractors”. You then take the percentage of customers who are “Promoters” and subtract the percentage who are “Detractors”. Obviously, our five-point scale does not perfectly fit this methodology, but let’s give it a try. To mimic NPS the best we can, we are going to use: (Very Satisfieds - (Somewhat Dissatisfieds + Very Dissatisfieds))/(Total number of respondents).
Obviously, each individual OEM’s absolute scores matter, and it is great if an OEM shows year-over-year improvement. But, clearly, it is NOT so great if others have improved MORE than you have. So, we not only care about absolute performance, but also relative performance – or an OEM’s RANK within the industry. Going back to our original question: Does the scoring methodology we use significantly impact the relative rank of an OEM? Let’s determine the rank of each OEM for each of the four scoring methodologies and look for significant differences in rank across methodologies.
In most cases, ranks differ by one or two positions; sometimes not at all. Bigger rank differences (>3) tend to occur for OEMs with smaller networks; scores will vary more in these cases.
The biggest difference in rank is 6, and it is for OEM 9 in the Service Manager Survey. Compared to OEMs below it, this OEM has more “Very Satisfieds” than “Very Satisfieds” plus “Somewhat Satisfieds”, respectively (which is a good thing). It is also the only case where the scoring makes the difference between “mid-pack” and “bottom of the barrel”. In general, the rank differences are greater in the Service Manager Survey, but at an average of 2.1 vs 1.4 for the Parts Manager Survey, this does not lead to significant shifts in rankings either way.
So, is this it: scoring methodologies do not really matter? Not so fast. Consider this picture:
Supply chain folks hate variability (they prefer the flat, steady, predictable lines); research folks love it. Simply put, where there is variability, there is “life”, and the opportunity to learn by figuring out where you stand and where you need to go. Thus, scoring methodologies tend to not significantly affect the ranking of survey participants, but they impact the score differences between participants, with some methodologies making the differences more visible and others, well, almost obliterating them.
Of course, applications do exist for methods with less variability. For instance, you may want to consider the Average or Top Two Box scoring methods if you tie performance targets and compensation elements to the survey scores. This is why we supply all box and raw data reports; participants can pursue the scoring methodology that best fits the purpose.
However, we prefer the Top Box method and use it as our official survey metric in North America. It acts as a “magnifying” glass to more clearly differentiate satisfaction performance. (As discussed, we don’t use NPS, as it is not really a satisfaction measurement.)
There are good reasons for our choice. When someone asks you “How are you doing?” you will most likely say “Good”, without even thinking about it. Most of us will not say “Very good” unless it has been a REALLY good day for us. Of course, this is what we REALLY want … and what we prefer over yet another boring regular, “good”, “flat line” day.
Take a look at the chart below from our 2013 Consumer Sentiment Survey, directed at vehicle owners. It shows the likelihood that a vehicle owner will return to the same dealer for service. The data is segmented by customers’ satisfaction with their most recent service event: 90% of the “Very Satisfieds” are “Very Likely” to go back to the same dealer vs. 63% of the merely “Satisfieds”. 27 points – quite a difference!
By the way, dealer personnel behave the same way as their customers do; they want to be “delighted” and not merely be “Satisfied”. Let’s take a look at parts manager purchase loyalty, which can be seen as equivalent to consumers’ “Service Repurchase Likelihood” in the prior chart.
The chart below is based on our 2013 Automotive Parts Manager Survey and cuts parts purchase loyalty by overall satisfaction level. There is a substantial difference between the purchase loyalty of “Very Satisfied” vs. “Satisfied” parts managers – almost 1.5 points. Multiply that by tens, hundreds or even billions in parts sales and you’ll see that this seemingly small difference matters … big time! And, because it does matter, our survey participants have decided to report satisfaction scores as “Top Box” (% “Very Satisfied”) instead of “Top Two Box” (% “Very Satisfied” plus % ”Satisfied”).
In other words, Top Box appropriately applies a stricter standard, by only looking at the “Very Satisfieds” who, conveniently, are also most likely the strongest promoters. (Note the similarity between the Top Box and NPS ranks in the tables above.) So, here is our answer to the question we posed above:
“Top Box or Top Two Box … or other metrics – that is the question.
We think 'tis nobler in the survey to score
By looking at Very Satisfieds only.”