When IT Disaster Strikes, Should ISVs Be Held Accountable?

Written by Evan Schuman
July 25th, 2008

When two unrelated IT disasters hit a major retailer and a major consumer goods manufacturer, executives for both firms this month pointed fingers at the two software companies involved: Microsoft and SAP.

But the comments made by top brass from Hannaford (hit by a 4.2-million-card data breach) and Levi’s (a troublesome ERP rollout suspended lots of shipments) have raised the question of where the responsibility line should be drawn. At what point is the software company responsible for doing what it has billed itself to do? And at what point is the CIO supposed to deliver, no matter what?

This is a question that isn’t going away, especially as more companies move increasingly toward an overwhelmingly outsourced enterprise environment, with an ever-shrinking percentage of apps that are homegrown. Sometimes, security assessors strongly encourage retailers to avoid homegrown applications.

In the Levi’s case, the problem was a massive ERP rollout, courtesy of SAP. During the implementation of SAP, Levi’s encountered issues and subsequently "chose to temporarily suspend shipments to our customers in the United States, causing us to miss the customer-requested delivery dates that fell within this period," according to an April 8 company SEC filing.

"This disruption lead to delays in filling customer orders and customer order cancelations that accounted for a substantial portion of our Q2 net revenue decline in the region," said Robert Hanson, president of the Americas region for Levi’s, in a conference call with reporters after the filing.

For Hannaford, the CIO at the time of this year’s data breach detailed some of the reasons for the breach and pointed at PCI shortcomings and security holes in Microsoft software.

"We used a lot of Linux," said Bill Homa, who stepped down as Hannaford CIO earlier this month. "None of the breach was anything related to Linux. All of it was Microsoft."

Added Homa: "Microsoft is so full of holes. That’s why it’s still a target. If you limit your exposure to Microsoft, you’re going to be in a more secure environment."

But in today’s complex IT environments, where does—and should—that line of responsibility fall? On the "it must stay with the CIO" argument comes the reality that a software glitch today is just as likely to involve a conflict with a competing app or an OS hiccup as some internal problem to that application. On the "it may stay with the ISV" argument is the fact that as large ISVs such as Oracle—in addition to Microsoft and SAP—continue to expand their product lines and promise end-to-end robustness, they must stand behind their work.

Paula Rosenblum, managing partner for Retail Systems Management and a former CIO herself, takes a decidedly pro-ISV perspective.

"The days are long since gone when a company can get up with a straight face and say, ‘I tried to do a major ERP implementation and it drove my business to its knees,’" Rosenblum said. "It just doesn’t work that way. The industry has to have matured to a point where that excuse is no longer valid. There was a period in the 90s when it was really popular."

Rosenblum said it’s expected that there will be bugs during a new ERP rollout, but she said "the art of managing a project and the art of implementing new systems isn’t about getting it 100 percent right. It’s about contingency planning. It’s about planning for all the stuff that can go wrong."

The comments by Hannaford’s Homa drew a larger than typical number of reader comments on the story, most of them also taking a decidedly pro-ISV (pro-Microsoft, in this case) position. Asked one anonymous reader: "He seems to be blaming (Microsoft) directly for the breach. I wonder if he ever signed a PO for an MS product during his tenure."

The missing factor here is what is reasonable to trust someone with and what isn’t? Last year, quite a few major retailers got caught up in some Fair and Accurate Credit Transactions Act (FACTA) lawsuits because they were illegally issuing consumers credit card receipts complete with full credit card numbers and expiration dates.

Many of those retailers argued that it was reasonable to have expected POS software vendors to deliver products that were consistent with federal law. I’d agree. Are we really expecting CIOs of $20 billion chains to test and verify the credit card printing apparatus? In hindsight, that’s reasonable, but at what point is it considered fiduciaryly (if it’s not a word, it should be) responsible to trust an application from a major software house?

That said, when we take this up quite a few notches and are talking about enterprise-wide ERP deployments or a top-level E-Commerce platform or payroll software, that equation is quite different.

Some have argued that the responsibility of the CIO doesn’t end with selecting the right vendors and having their products extensively tested. It also includes making the assumptions that most things will indeed fail—Murphy’s IT law: the more strategically important the rollout, the better the chance it will fail—and to prepare backup plans.

That’s a fair point. But there is a huge difference between reckless indifference and making a reasonable decision and implementing reasonable backups that happened to prove inadequate.

The question boards must ask: "Given what the CIO knew at the time, was the action (or inaction) reasonable, especially considering the finite budget we have granted that CIO?" Better yet, let’s pose that question in the reverse: "If the CIO had spent $XX million dollars on an additional redundant (a Monday Morning Quarterback might say "unnecessary") system and the deployment went fine, would we have been justly angry with that CIO? Honestly?"

After a major IT disaster, it’s easy to point the finger at a major ISV or industry guidelines, especially when there’s good cause to do so. Is anyone going to argue that Microsoft’s OS is truly secure? Or that PCI could stand a lot of improvement? Or that SAP ERP deployments aren’t always trouble free?

But it’s just as easy to point the finger at the CIO—after the fact—and say, "You should have anticipated this and avoided this problem."

But if both sides tried less blaming and more reasonable balance, it couldn’t hurt.


2 Comments | Read When IT Disaster Strikes, Should ISVs Be Held Accountable?

  1. Paula Rosenblum Says:

    God forbid I should be accused of pandering to the vendors – let me take a moment to fully spell out what I said, as my remarks were taken slightly (very slightly) out of context.

    I said something to the effect of, “Whether it’s Oracle, SAP or any other large software vendor, there are many, many cases where the application went in fairly smoothly. All these applications have matured to the point where are they are widely used. Implementation glitches are to be expected. Problems typically come down to 2 issues: 1) Lack of contingency planning and 2) organizational willingness to change. Any organization that has the kind of tangled systems Levi self-acknowledged when it decided to roll out SAP probably also has amazingly tangled procedures as well. It wouldn’t surprise me if pieces of the ‘is’ was not documented into the ‘will be’. There was also a Sarbanes-Oxley issue under the covers as well. A lot of muttering in the filing about internal controls. That goes to the procedures…not the application.”

    I know nothing more about the situation at LS&CO than that.

    And, to be very very clear — I rarely would blame the CIO. That’s just wrong. I am much more inclined to blame the user community in the case of an ERP implementation (I AM a former CIO after all!!).

    The Homa story? I have no opinion on it. There’s just bad blood there, as far as I can see. Hannaford got breached. It’s unfortunate, but these things happen. PCI was never enough and the industry’s obsessive focus on it alone was a mistake. Data security is a fluid and incredibly challenging issue.

    Hope that clarifies my remarks.

  2. Erik Bergeman Says:

    Wanted to add a comment in response to the following paragraph: “Many of those retailers argued that it was reasonable to have expected POS software vendors to deliver products that were consistent with federal law. I’d agree. Are we really expecting CIOs of $20 billion chains to test and verify the credit card printing apparatus? In hindsight, that’s reasonable, but at what point is it considered fiduciaryly (if it’s not a word, it should be) responsible to trust an application from a major software house?” This argument has a tendancy to really fall down when the retailer implements a version of POS 15 or 20 years old and won’t upgrade because of cost. Many times the retailer is using a version 5 or 10 versions back of an ISV’s current application version. Because these regulations and breaches have been developed over time some of the older versions will still have the holes that can be exploited. How do you hold the ISV responsible when the CIO has not upgraded to the latest version that has fixed many if not all the problems?


StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

Most Recent Comments

Why Did Gonzales Hackers Like European Cards So Much Better?

I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
@David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.