Evan Schuman's StorefrontBacktalk

Shoppers were asked to rate experiences from the prior 60 days. “While consumers rated many companies, we limited our analysis to only those 246 companies across 19 industries for which we had responses from at least 100 consumers.”

When it comes to online survey methodology, Tempkin Group’s approach sounds as good as any. But it suffers from the same contextual problems as similar surveys. Assuming that these people are self-reporting their specs accurately and that no one is trying to game the system, the survey is still dependent on frail memories of shoppers about experiences during the prior two months.

Those recollections can be colored by quite a few things. Let’s say someone has to wait a long time in an Apple Store to get an associate’s attention and then the associate couldn’t answer several key questions—or answered them incorrectly. As a result, the wrong unit was brought and other customer service problems materialized. But the product worked great at home, and the shopper is now in love with her iPad. Those favorable feelings are very likely to color her recollections and, hence, her answers. The same thing works in reverse. If you’re Target (NYSE:TGT) and you’re trying to improve your customer service performance, you want to know customers’ reactions on the day of the purchase, which is when store customer service is likely to be the key factor in their assessments.

Questions need to differentiate between what store associates are doing to help the sales process (and certainly post-sales assistance can be key) and customers’ feelings about the product purchased two months later. That said, this particular battle of reality versus perception may be won by perception. If Costco (NASDAQ:COST) customers in March think they had a bad customer experience in a store back in January, it doesn’t

matter whether they really did. Their behavior will be dictated by their perceptions and recollections.

The real problems with reports like this—and the Tempkin Group offering appears to be one of the better entries—is that the answers are not nearly specific enough to be actionable. For example, why did Apple do poorly and Ace Hardware perform so well? Without follow-up questions that delve into specifics, the datapoints are interesting but, ultimately, not that useful.

Tempkin dubbed the performance of both Apple and Ace as “surprises,” but he could only guess as to the reasons. “Ace Hardware was a surprise, but I think the store franchise structure leads to more personal service. I think the interesting item is comparing Sam’s Club to Nordstrom. The lesson is that customer experience is about consistently living up to brand promises, not about trying to replicate how other companies deliver their experience. Sam’s Club does that well,” Tempkin said, adding that Apple’s surprise might be explained by unmet expectations. “It could also be that Apple has a very avid fan base in some segments and not so much in others.”