Evan Schuman's StorefrontBacktalk

After initially suspecting that one of its third-party payment card processors had caused a December 20 mess where some 8,000 Macy’s customers had their debit cards charged as many as three times for one transaction, the chain’s payment management has now concluded that the fault was solely within their internal software.

“We were looking at the processor and the bank networks, trying to determine whether this was an issue with specific banks,” Mike Gatio, president of Macy’s credit and customer services division, said in a Wednesday (Jan. 14) interview. “We’ve now narrowed it down to our own gateway. We deal with a number of processors, but it’s not their issue. It’s ours.”

That is a reversal within Macy’s. On Monday (Jan. 12) evening, when StorefrontBacktalk caught up with Macy’s CEO Terry Lundgren at the NRF show, he put the blame squarely on one of the chain’s outside processors, but said the details should come from others within the company who were closer to the incident.

Although a gateway that takes debit card transactions from Macy’s POS network and coordinates them before sending them to various payment card processors had problems on December 20 due to excessive traffic, Gatio said the greatest failure came from a different piece of software. That application had the sole function of watching for identical sales tickets that are processed multiple times and automatically issuing credits for all but one of the transactions.

“In our system, we have a function that when a transaction goes to a bank and doesn’t come back, we are supposed to automatically flag that and automatically credit the customer’s account,” Gatio said. “But in that period, for whatever reason, that function didn’t operate correctly. We are investigating now how that happened.”

The December 20 debit card glitch impacted in-store shoppers only—anyone who used a debit card to make online purchases was spared—and it was also limited to PIN debit transactions, ignoring signature debit transactions. It impacted hundreds of Macy’s stores in parts of Alabama, Georgia, Kentucky, Louisiana, North Carolina, Oklahoma, South Carolina, Tennessee, Texas, Virginia, West Virginia, Illinois, Indiana, Kansas, Missouri, New York, Ohio, Pennsylvania, Michigan, Minnesota, North Dakota, Ohio, South Dakota and Wisconsin.

Exactly What Happened

The situation began December 20 at about 1 PM New York time, at the height of an especially transaction-intensive day. As debit transactions “got bottled up in the gateway process,” said Jim Sluzewski, Macy’s VP for corporate communications, transactions were getting approved, “but the signal back from the bank saying the transaction was approved didn’t get recorded as a return message.”

Store associates at POS stations would then ask the customer to swipe the debit card again. “In a small handful of those, probably 20 or so, the associate said, ‘It didn’t go through again. Do you want to try that one more time?’,” an action that delivered the incident’s few triple charges. “Most people gave up after two attempts” and offered some other payment method, Sluzewski said.

Some internal systems then started noticing the problem. “We have an internal system that picks up aberrations. It started to flag the fact that there were some potential double debits happening,” Sluzewski said. “At that point, we started diagnosing the problem, trying to figure out where the issue was. At 1:30 to 1:45, we started shifting stores to an alternate gateway. By 2:45, all stores had been shifted to an alternate gateway.”

Given that all Macy’s locations were likely similarly overloaded, Gatio said that shifting hundreds of stores to an already-used gateway was not without its risks. “Could we have potentially overwhelmed the alternate gateway? Yeah. We tracked this in real-time. There was a higher sensitivity, I’m sure, between our systems team in Atlanta and our group.”

Gatio’s team is trying to establish why both the gateway—and the internal verification system—erred at the same time. “The gateway backed up. The transactions were getting out, but confirmations weren’t coming back in. Our internal system at Macy’s is programmed to notice, to see that that’s happening. And if there is a double debit in that process, it’s supposed to automatically send a credit for the second debit. In that period of time, that bounce-back didn’t happen.”

“The immediate concern, from our vantage point, is why didn’t the automated reversal work as designed. It may be because it was overwhelmed. We don’t know that,” Gatio said. “That automated reversal has to take place. It should take place and it typically does. We’ve got to figure out why that failed.”

What Went Wrong

As for the gateway’s collapse, Gatio said that is a bit easier to figure out. “This obviously was the largest Saturday of the year. Clearly, the gateway, from our perspective, was overwhelmed just due to sheer volume. The network, the whole communication, between us and the acquirer, was responding slower,” Gatio said. “It probably was a combination of two things going on. We know from our vantage point that the volumes (were high) but this was more of an issue of the gateway capacity, not store traffic. It’s likely the gateway configuration we had for that particular gateway for that day wasn’t sufficient. The volume overwhelmed the gateway.”

“We need to postmortem this. Did we have the proper configuration at each gateway to handle the busiest Saturday of the year?” Gatio said. “We cannot afford to have that automated reversal process fail. It should be foolproof.”

Macy’s officials also gave their side of the statement issued by the Connecticut attorney general on Jan. 9 that his office was investigating the debit card issue in connection with some Connecticut residents who were apparently impacted.

Sluzewski said that Macy’s was told by Connecticut authorities about only a single customer that had been double billed on December 24, long after, according to Macy’s, its debit problem had been repaired.

“We know we didn’t have any gateway issues” at that time, Gatio said. “It was not an internal gateway incident with Macy’s at all.”

Both Sluzewski and Gatio said the Connecticut AG’s office didn’t speak with them until after it had announced the statement.

Connecticut Attorney General Richard Blumenthal, in a phone interview shortly after the Macy’s executives spoke, confirmed that his office and Macy’s hadn’t spoken until after the statement had been issued. But he said that it wasn’t because of a lack of trying on Connecticut’s part. “It took awhile to get their attention,” Blumenthal said.

Blumenthal denied that the state only told Macy’s about one incident.

“We have an ongoing investigation. There is no evidence at this point that the problem is widespread. We’ve received a number of contacts from people with complaints,” Blumenthal said, “but we can’t say at this point whether it’s the same problem as occurred in the Midwest or a number of isolated incidents, and we’re continuing our investigation. I’m not telling you what the number is.”

Given that one and zero are both numbers, the AG’s comments don’t shed much light.