Macy’s Own Software Caused Its Holiday Debit Card Glitch

Written by Evan Schuman and Fred J. Aun
January 14th, 2009

After initially suspecting that one of its third-party payment card processors had caused a December 20 mess where some 8,000 Macy’s customers had their debit cards charged as many as three times for one transaction, the chain’s payment management has now concluded that the fault was solely within their internal software.

“We were looking at the processor and the bank networks, trying to determine whether this was an issue with specific banks,” Mike Gatio, president of Macy’s credit and customer services division, said in a Wednesday (Jan. 14) interview. “We’ve now narrowed it down to our own gateway. We deal with a number of processors, but it’s not their issue. It’s ours.”

That is a reversal within Macy’s. On Monday (Jan. 12) evening, when StorefrontBacktalk caught up with Macy’s CEO Terry Lundgren at the NRF show, he put the blame squarely on one of the chain’s outside processors, but said the details should come from others within the company who were closer to the incident.

Although a gateway that takes debit card transactions from Macy’s POS network and coordinates them before sending them to various payment card processors had problems on December 20 due to excessive traffic, Gatio said the greatest failure came from a different piece of software. That application had the sole function of watching for identical sales tickets that are processed multiple times and automatically issuing credits for all but one of the transactions.

“In our system, we have a function that when a transaction goes to a bank and doesn’t come back, we are supposed to automatically flag that and automatically credit the customer’s account,” Gatio said. “But in that period, for whatever reason, that function didn’t operate correctly. We are investigating now how that happened.”

The December 20 debit card glitch impacted in-store shoppers only—anyone who used a debit card to make online purchases was spared—and it was also limited to PIN debit transactions, ignoring signature debit transactions. It impacted hundreds of Macy’s stores in parts of Alabama, Georgia, Kentucky, Louisiana, North Carolina, Oklahoma, South Carolina, Tennessee, Texas, Virginia, West Virginia, Illinois, Indiana, Kansas, Missouri, New York, Ohio, Pennsylvania, Michigan, Minnesota, North Dakota, Ohio, South Dakota and Wisconsin.

Exactly What Happened

The situation began December 20 at about 1 PM New York time, at the height of an especially transaction-intensive day. As debit transactions “got bottled up in the gateway process,” said Jim Sluzewski, Macy’s VP for corporate communications, transactions were getting approved, “but the signal back from the bank saying the transaction was approved didn’t get recorded as a return message.”

Store associates at POS stations would then ask the customer to swipe the debit card again. “In a small handful of those, probably 20 or so, the associate said, ‘It didn’t go through again. Do you want to try that one more time?’,” an action that delivered the incident’s few triple charges. “Most people gave up after two attempts” and offered some other payment method, Sluzewski said.

Some internal systems then started noticing the problem. “We have an internal system that picks up aberrations. It started to flag the fact that there were some potential double debits happening,” Sluzewski said. “At that point, we started diagnosing the problem, trying to figure out where the issue was. At 1:30 to 1:45, we started shifting stores to an alternate gateway. By 2:45, all stores had been shifted to an alternate gateway.”

Given that all Macy’s locations were likely similarly overloaded, Gatio said that shifting hundreds of stores to an already-used gateway was not without its risks. “Could we have potentially overwhelmed the alternate gateway? Yeah. We tracked this in real-time. There was a higher sensitivity, I’m sure, between our systems team in Atlanta and our group.”

Gatio’s team is trying to establish why both the gateway—and the internal verification system—erred at the same time. “The gateway backed up. The transactions were getting out, but confirmations weren’t coming back in. Our internal system at Macy’s is programmed to notice, to see that that’s happening. And if there is a double debit in that process, it’s supposed to automatically send a credit for the second debit. In that period of time, that bounce-back didn’t happen.”

“The immediate concern, from our vantage point, is why didn’t the automated reversal work as designed. It may be because it was overwhelmed. We don’t know that,” Gatio said. “That automated reversal has to take place. It should take place and it typically does. We’ve got to figure out why that failed.”

What Went Wrong

As for the gateway’s collapse, Gatio said that is a bit easier to figure out. “This obviously was the largest Saturday of the year. Clearly, the gateway, from our perspective, was overwhelmed just due to sheer volume. The network, the whole communication, between us and the acquirer, was responding slower,” Gatio said. “It probably was a combination of two things going on. We know from our vantage point that the volumes (were high) but this was more of an issue of the gateway capacity, not store traffic. It’s likely the gateway configuration we had for that particular gateway for that day wasn’t sufficient. The volume overwhelmed the gateway.”

“We need to postmortem this. Did we have the proper configuration at each gateway to handle the busiest Saturday of the year?” Gatio said. “We cannot afford to have that automated reversal process fail. It should be foolproof.”

Macy’s officials also gave their side of the statement issued by the Connecticut attorney general on Jan. 9 that his office was investigating the debit card issue in connection with some Connecticut residents who were apparently impacted.

Sluzewski said that Macy’s was told by Connecticut authorities about only a single customer that had been double billed on December 24, long after, according to Macy’s, its debit problem had been repaired.

“We know we didn’t have any gateway issues” at that time, Gatio said. “It was not an internal gateway incident with Macy’s at all.”

Both Sluzewski and Gatio said the Connecticut AG’s office didn’t speak with them until after it had announced the statement.

Connecticut Attorney General Richard Blumenthal, in a phone interview shortly after the Macy’s executives spoke, confirmed that his office and Macy’s hadn’t spoken until after the statement had been issued. But he said that it wasn’t because of a lack of trying on Connecticut’s part. “It took awhile to get their attention,” Blumenthal said.

Blumenthal denied that the state only told Macy’s about one incident.

“We have an ongoing investigation. There is no evidence at this point that the problem is widespread. We’ve received a number of contacts from people with complaints,” Blumenthal said, “but we can’t say at this point whether it’s the same problem as occurred in the Midwest or a number of isolated incidents, and we’re continuing our investigation. I’m not telling you what the number is.”

Given that one and zero are both numbers, the AG’s comments don’t shed much light.


One Comment | Read Macy’s Own Software Caused Its Holiday Debit Card Glitch

  1. Ward Cleaver Says:

    Heartland is too big with too many merchants for Visa and MC to beat-up on and threaten to cut them out of the system as they did to Card Systems in 2005.

    Heartland itself has bullied alot of merchants over PCI, has . Now the bully takes a punch to the nose.


StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

Most Recent Comments

Why Did Gonzales Hackers Like European Cards So Much Better?

I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
@David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.