$12 Million In Duplicate Charges From Shell Oil Telco Crash
Written by Evan SchumanOn Saturday (Jan. 29), a telco outage at Shell Oil stations directly caused more than $12 million in duplicate charges for its retail customers. This is yet the latest case of a chain getting burned because the payment industry has no consistent way to deal with in-progress credit and debit charges when systems crash. Does store-and-forward need to be tweaked to be made more crash-proof?
First Data circulated a confidential memo on Monday (Jan. 31) that said it was reversing some 401,120 transactions—totaling $12,135,608.19—of Shell’s retail transactions from the weekend. First Data asked issuers to “post the reversals to the cardholder accounts impacted as soon as possible in order to limit negative impact.” (Guess “negative impact” means customers screaming at call center reps.)
(Important Story Update: AT&T is now denying that any outage took place on their network and Shell is quickly backtracking. See the latest here.)
The incident happened on Saturday (Jan. 29) when Shell “experienced a system issue that impacted credit and debit card processing,” said Shell spokesperson Theodore Rolfvondenbaumen. The credits were mostly made Tuesday (Feb. 1), he said.
Although the details are still being investigated, Rolfvondenbaumen said it was an AT&T “unplanned telecommunications outage” (he conceded that he’s never seen a planned telco outage) and that AT&T handles Shell’s payment card processing network connections.
“Our IT department is investigating the root cause,” he said. “This is the first time it’s ever happened in the Shell network.”
Among those more than 401,000 transactions were “debit card customers who did experience overdraft fees” along with many credit card customers. Shell is “willing to work directly with debit card customers” to cover the costs of overdraft and related fees. “If they were impacted, if someone has been impacted, we will work to make this right for them,” Rolfvondenbaumen said.
This is yet another reminder of the high-cost of retailers accepting debit cards. The interchange fees are lower, but the cards are also much more susceptible to glitching. And when problems happen, the costs are so much higher than for credit cards, as Shell has discovered. Retailers are perhaps too quick to dismiss the retail benefits of zero-liability programs.
The bigger issue, though, is how a telco outage could have caused so many duplicate charges.
-Marc
February 3rd, 2011 at 10:39 am
Amazing that people and other businesses never pass on an opportunity to take a shot at AT&T. This problem was caused not by the Telco outage, but rather the flawed First Data Software code. Network outages occur, it is a fact of life in a wired network there will always be outages. Apparently Shell and First Data did not contemplate that an outage might occur and what would happen if it did.
February 3rd, 2011 at 12:42 pm
The thing that is so crazy about this story is that it is so simple to prevent. Every transaction initiated requires a unique ID. Whether the ID is explicitly stated or calculated by hashing values in the transaction, the ID provides a means for screening duplicates.
But this must be an age old problem, because I can remember the VP of Tech Services reviewing transaction dumps to manually screen duplicates. We operated our own “switch” and routed the credit card transactions coming in from the stores to the providers. I never got an explanation why this couldn’t be programmed back then, but apparently it still exists.
February 3rd, 2011 at 1:48 pm
In dealing with issues like this for over 20 years and designing systems and subsystems to prevent or minimize issue like this, I don’t blame any of the parties mentioned in the article. Instead, I blame a fundamental flaw in the core authorization/clearing systems run by the card brands. The one piece missing is an end-to-end Globally Unique ID (GUID) assigned to the transaction at the point of origin. This point of origin GUID assignment should be as close to the point of entry as possible (or point of initiation in the event of recurring transaction).
Right now there is no such capability and without it, you can write all the preventative code you want but at some point a decision needs to be made — error in favor of the cardholder, or error in favor of the merchant and arguments could be made for both. Adding this GUID would (or should) guarantee no double postings no matter how many times the transaction is retried or reposted.
As to the liability argument, good luck. The only winners will be the lawyers. Even SLA’s won’t stop the finger pointing: the POS vendor should have coded for this, the merchant should have audited prior to settlement, a velocity threshold should have alerted First Data, etc., the possibilities are endless.
February 3rd, 2011 at 6:59 pm
This also happened on Friday January 28 in the afternoon as I experienced the double bill from a Shell gas transaction around 4:30pm CST 01-28-11.
March 23rd, 2011 at 11:18 pm
Absence of uniqueness has been a huge question mark & I thought that this has been a black box to me as I did not get clear answers. Thanks to Steve Sommers on throwing light. There are many ways by which this can be achieved. As we move towards chip & pin transactions, one quick suggestion could be card number combined with card transaction counter (CTC). Would ISO include this or other as an enhancement to 8583? This uniqueness will make a lot of sense to payment industry.