$12 Million In Duplicate Charges From Shell Oil Telco Crash

Written by Evan Schuman
February 2nd, 2011

On Saturday (Jan. 29), a telco outage at Shell Oil stations directly caused more than $12 million in duplicate charges for its retail customers. This is yet the latest case of a chain getting burned because the payment industry has no consistent way to deal with in-progress credit and debit charges when systems crash. Does store-and-forward need to be tweaked to be made more crash-proof?

First Data circulated a confidential memo on Monday (Jan. 31) that said it was reversing some 401,120 transactions—totaling $12,135,608.19—of Shell’s retail transactions from the weekend. First Data asked issuers to “post the reversals to the cardholder accounts impacted as soon as possible in order to limit negative impact.” (Guess “negative impact” means customers screaming at call center reps.)

(Important Story Update: AT&T is now denying that any outage took place on their network and Shell is quickly backtracking. See the latest here.)

The incident happened on Saturday (Jan. 29) when Shell “experienced a system issue that impacted credit and debit card processing,” said Shell spokesperson Theodore Rolfvondenbaumen. The credits were mostly made Tuesday (Feb. 1), he said.

Although the details are still being investigated, Rolfvondenbaumen said it was an AT&T “unplanned telecommunications outage” (he conceded that he’s never seen a planned telco outage) and that AT&T handles Shell’s payment card processing network connections.

“Our IT department is investigating the root cause,” he said. “This is the first time it’s ever happened in the Shell network.”

Among those more than 401,000 transactions were “debit card customers who did experience overdraft fees” along with many credit card customers. Shell is “willing to work directly with debit card customers” to cover the costs of overdraft and related fees. “If they were impacted, if someone has been impacted, we will work to make this right for them,” Rolfvondenbaumen said.

This is yet another reminder of the high-cost of retailers accepting debit cards. The interchange fees are lower, but the cards are also much more susceptible to glitching. And when problems happen, the costs are so much higher than for credit cards, as Shell has discovered. Retailers are perhaps too quick to dismiss the retail benefits of zero-liability programs.

The bigger issue, though, is how a telco outage could have caused so many duplicate charges.


5 Comments | Read $12 Million In Duplicate Charges From Shell Oil Telco Crash

  1. Jeff Says:

    Amazing that people and other businesses never pass on an opportunity to take a shot at AT&T. This problem was caused not by the Telco outage, but rather the flawed First Data Software code. Network outages occur, it is a fact of life in a wired network there will always be outages. Apparently Shell and First Data did not contemplate that an outage might occur and what would happen if it did.

  2. bill bittner Says:

    The thing that is so crazy about this story is that it is so simple to prevent. Every transaction initiated requires a unique ID. Whether the ID is explicitly stated or calculated by hashing values in the transaction, the ID provides a means for screening duplicates.

    But this must be an age old problem, because I can remember the VP of Tech Services reviewing transaction dumps to manually screen duplicates. We operated our own “switch” and routed the credit card transactions coming in from the stores to the providers. I never got an explanation why this couldn’t be programmed back then, but apparently it still exists.

  3. Steve Sommers Says:

    In dealing with issues like this for over 20 years and designing systems and subsystems to prevent or minimize issue like this, I don’t blame any of the parties mentioned in the article. Instead, I blame a fundamental flaw in the core authorization/clearing systems run by the card brands. The one piece missing is an end-to-end Globally Unique ID (GUID) assigned to the transaction at the point of origin. This point of origin GUID assignment should be as close to the point of entry as possible (or point of initiation in the event of recurring transaction).

    Right now there is no such capability and without it, you can write all the preventative code you want but at some point a decision needs to be made — error in favor of the cardholder, or error in favor of the merchant and arguments could be made for both. Adding this GUID would (or should) guarantee no double postings no matter how many times the transaction is retried or reposted.

    As to the liability argument, good luck. The only winners will be the lawyers. Even SLA’s won’t stop the finger pointing: the POS vendor should have coded for this, the merchant should have audited prior to settlement, a velocity threshold should have alerted First Data, etc., the possibilities are endless.

  4. econobiker Says:

    This also happened on Friday January 28 in the afternoon as I experienced the double bill from a Shell gas transaction around 4:30pm CST 01-28-11.

  5. Malathi Says:

    Absence of uniqueness has been a huge question mark & I thought that this has been a black box to me as I did not get clear answers. Thanks to Steve Sommers on throwing light. There are many ways by which this can be achieved. As we move towards chip & pin transactions, one quick suggestion could be card number combined with card transaction counter (CTC). Would ISO include this or other as an enhancement to 8583? This uniqueness will make a lot of sense to payment industry.


StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

Most Recent Comments

Why Did Gonzales Hackers Like European Cards So Much Better?

I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
@David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.