Tuesday, January 27, 2009

Why J.D. Power CSI Is Stupid, or It Hurts to Be In Love – The Formula

Many consumers fell in love with the power of customer satisfaction in response to one of the many 1980’s apocryphal stories of how service sells. A favorite story concerns an irate woman at Nordstrom who wanted to return a set of tires her husband bought her for Christmas. The salesperson was understanding and processed a store credit on the spot. But, Nordstrom does not sell tires. Wow. One set of 1980’s phenoms were David Power and David Halberstam. The Power David launched the first Consumer Satisfaction Index (CSI) in 1981, and The Reckoning David published The Reckoning in 1986. Book and index coupled to make another message on the power of quality and satisfaction. Nissan was coroneted, Ford debauched, and quality was king. We were in love.

Satisfaction = Success. That’s The Formula.

Due to all the press the Power CSI gets, consumers are significantly influenced by their scorecards and awards. OK, so just using data published for free on the internet, let’s take on the role of customer advocate and check J.D. Power CSI against the critical “is it stupid” test … and see if we are still in love. Remember, from the internet (Dictionary.com), Stupid means: lacking ordinary quickness and keenness of mind; characterized by or proceeding from mental dullness; foolish; tediously dull, esp. due to lack of meaning or sense; inane; pointless: a stupid party; annoying or irritating; troublesome. JD Power releases their CSI scores to the public on their web site, and you can get motor vehicle sales from a bunch of places – Automotive News is the best site for this . To check out The Formula, last year’s car wreck of an industry would be a good proving ground. Theoretically, OEMs with higher levels of satisfaction should have been the least hurt.

Well, that certainly didn’t happen. When we charted the CSI numbers versus volume change we couldn’t see anything. Looks like the CSI indexes are all in the same band

and there is little differentiation between the OEMs. Hmm. Maybe we need some arcane transcendental statistical technique to sort the goodies from the baddies here? Certainly, that explains why Joe and Joanne the Plumber can use this sort of data to help them pick a car or truck. For example, we could have collapsed the scale on the CSI index and showed the trend in the 750 – 1,000 point band. (This is what we call the Alaskan “Joe 6-Pack Chart”, you know, where Joe/Josephine charts decibel-ized TV-room responses during the Super bowl and concludes that beer and TV correlate with inner feelings, sensitivity, care, and tenderness – it’s the little chart sitting in the lower right corner of the big chart above.) Nah, nothing there.

Here’s another way of looking at these numbers. This chart below plots the change in OEM volume from 2007 to 2008 versus the change in JD Power CSI. The theory here is that change in satisfaction is the real explanation behind The Formula. Nah! The OEM who did the best, volume-wise, actually went down in CSI. There is no visual pattern here that supports The Formula.

OK, we need to keep looking.
In the chart above we plotted CSI changes versus market share changes from 2007 to 2008. The OEM logos on the circumference of the scatter plot are sitting near or behind their data points. OK, there’s no visual correlation between satisfaction changes and share changes, but the plot does confirm Bob Lutz’s first three laws of automotive success: (1) PRODUCT, (2) PRODUCT, (3) PRODUCT. Happy Happy OEMs had great small car product; Happy Shrinkers had too much truck and SUVs in their product lineup; Unhappy Prosperity OEMs had product that the recessionary market really wanted; and, Sad Sacks were truck makers and yesterday’s The Formula poster children. Hmm. Looking back a decade ago, Toyota was above average in CSI, the highest score was not over 200, and the range of high to low was more than 2 to 1. Now, Toyota is below average in CSI, there is a 1000 point scale, and the spreads of the scores is much narrower. Does this infer that the secret to Toyota’s success is something they learned from interrogation techniques practiced by 24’s Jack Bauer?

Does this mean that The Formula is kaput? No. We need to use our commonsense on this one. It is human nature to spend more time with people and services that make us satisfied than with those people and services that make us unsatisfied. This works for restaurants, hardware stores, and motor vehicle companies. So we know that The Formulas is real. So, why can’t we prove this with all this brand name data? It probably means that the brand name measurement of customer satisfaction measures something else. But it is no longer a helpful tool in achieving success though real, tangible, and relevant customer satisfaction. My guess is that there are loads of senior executives at the OEMs who don’t know this. Otherwise, they’d be taking out their cost cutting hatchet and lopping off some dead limbs. Many missed the ball over the past decade and started to compete in J.D. Power’s backyard of abstract CSI scoring rather than, quite brilliantly, like Toyota in the showroom.

So, what can you do with all this?
We need to be stronger advocates for change when we see something stupid that costs us a lot of money and makes us behave irrationally. In terms of CSI, we need something different - something that works. We need something that measures “satisfaction” attributes explaining why customers do not buy stuff from dealers and why they defect.
  • We need satisfaction data from a valid sample of customers who aren’t paid a dollar to fill impossibly long surveys.

  • We need to have a balanced sample of customers that does not overly represent folks who are angry and have an axe to grind.

  • We need to ask the fewest possible questions on the survey that accurately represents their satisfaction and gets to why they are not satisfied.

  • We need to ask customers if they are going to continue to come back to dealers for service and, if they say no, we need to understand why in actionable terms.

  • We need a process that is responsible to the “whys” of the scores. For example, why is Toyota so successful while losing competitive ground in CSI over the past 10 years?

OK, so if you have the internet, love data, and don’t mind questioning “common knowledge”, it really doesn’t hurt to be in love. It allows you to fess out those Jezebel concepts out there. Hey, it’s a recession and now’s the time to clean up shop and not merely follow the crowd to the slaughterhouse.

Question for Blog Readers from David Carlisle: It takes us about 40 man-hours each week to publish these blogs, and we need to know if it is worth it. Do they make a difference? Are they helpful? What should we be doing with new topics? What should we be doing differently? I have very thick skin, so I can handle tough feedback. Please take a few minutes to tell me how we are doing. Click here to email me feedback. david.carlisle@carlisle-co.com

Note: All of the CSI data used in the analyses in this blog were published by J.D. Power and Associates on their web site, which has free public access, and/or downloaded from internet newspaper sites.

No comments: