This is page 2 of:
$12 Million In Duplicate Charges From Shell Oil Telco Crash
The bigger issue, though, is how a telco outage could have caused so many duplicate charges. Typically, duplicate charges happen when store employees are dealing with transactions that were in progress at the instant when the outage hit. If the system records are unclear as to whether the charge was fully processed and confirmed or not, associates (and, more likely, the system’s automatic process will kick in) will typically take the safe route—in terms of revenue protection—and process the charges again.
Either way, as Shell discovered, this approach can cause a lot of false charges. And it is painful well beyond the accounting cleanup. Beyond the cost of processing the reversals—and hard dollar reimbursement costs, such as for those debit customers who were hit with overdraft fees—there’s the hit to the chain’s reputation. How will customers who see such double charges feel? Will they assume the chain was trying to rip them off?
This possibility also raises some fun questions. Can Shell legal connect the dots sufficiently to make the case to AT&T? And if so, can they get AT&T to cover all of the losses in any way associated with the outage? Should it be able to?
But let’s expand this discussion. Outages that can trigger this store-and-forward headache are many, from power blackouts to telco hiccups (such as what Shell experienced) to—for E-Commerce transactions—a Web site crash or network disconnect or even a backbone provider problem.
Getting back to the AT&T liability issue, should this now be a standard part of performance guarantees? Sort of an SLA (service-level agreement) type of agreement that covers companies of this type?
The short answer is almost certainly not, because the AT&Ts and others are going to legitimately say that the duplicates weren’t really caused by the outage but by a problem with the store-and-forward system that they have nothing to do with.
The problem is that store-and-forward systems are supposed to be able to identify those transactions that had been properly processed and those that hadn’t. With more than 401,000 dupes coming from as large a chain as Shell, isn’t there a fundamental flaw?
This isn’t the first time First Data has been involved in a double-charge problem. It experienced one with Hannaford and other chains less than four months ago. In that case, customers were not only double-billed, but the retailers were double-paid. The day before Thanksgiving (Nov. 24, 2010), grocery chain Winn-Dixie—suffering an outage—reported double-charging customers in all of its 485 stores.
There have also been cases of a million double-charged transactions at Starbucks as well as publicized incidents at Macy’s plus a big one at Best Buy.
-Marc
February 3rd, 2011 at 10:39 am
Amazing that people and other businesses never pass on an opportunity to take a shot at AT&T. This problem was caused not by the Telco outage, but rather the flawed First Data Software code. Network outages occur, it is a fact of life in a wired network there will always be outages. Apparently Shell and First Data did not contemplate that an outage might occur and what would happen if it did.
February 3rd, 2011 at 12:42 pm
The thing that is so crazy about this story is that it is so simple to prevent. Every transaction initiated requires a unique ID. Whether the ID is explicitly stated or calculated by hashing values in the transaction, the ID provides a means for screening duplicates.
But this must be an age old problem, because I can remember the VP of Tech Services reviewing transaction dumps to manually screen duplicates. We operated our own “switch” and routed the credit card transactions coming in from the stores to the providers. I never got an explanation why this couldn’t be programmed back then, but apparently it still exists.
February 3rd, 2011 at 1:48 pm
In dealing with issues like this for over 20 years and designing systems and subsystems to prevent or minimize issue like this, I don’t blame any of the parties mentioned in the article. Instead, I blame a fundamental flaw in the core authorization/clearing systems run by the card brands. The one piece missing is an end-to-end Globally Unique ID (GUID) assigned to the transaction at the point of origin. This point of origin GUID assignment should be as close to the point of entry as possible (or point of initiation in the event of recurring transaction).
Right now there is no such capability and without it, you can write all the preventative code you want but at some point a decision needs to be made — error in favor of the cardholder, or error in favor of the merchant and arguments could be made for both. Adding this GUID would (or should) guarantee no double postings no matter how many times the transaction is retried or reposted.
As to the liability argument, good luck. The only winners will be the lawyers. Even SLA’s won’t stop the finger pointing: the POS vendor should have coded for this, the merchant should have audited prior to settlement, a velocity threshold should have alerted First Data, etc., the possibilities are endless.
February 3rd, 2011 at 6:59 pm
This also happened on Friday January 28 in the afternoon as I experienced the double bill from a Shell gas transaction around 4:30pm CST 01-28-11.
March 23rd, 2011 at 11:18 pm
Absence of uniqueness has been a huge question mark & I thought that this has been a black box to me as I did not get clear answers. Thanks to Steve Sommers on throwing light. There are many ways by which this can be achieved. As we move towards chip & pin transactions, one quick suggestion could be card number combined with card transaction counter (CTC). Would ISO include this or other as an enhancement to 8583? This uniqueness will make a lot of sense to payment industry.