PCI’s Grading System Is Failing

Written by David Taylor
April 29th, 2009

GuestView Columnist David Taylor is the Founder of the PCI Knowledge Base, Research Director of the PCI Alliance and a former E-Commerce and Security analyst with Gartner.

For months, retailers and Congress have been attacking retail security standards, but few realize that the problem is not in the standard itself. The problem is a grading system that causes most retailers to be out of compliance most of the time because the rules require 100 percent compliance. How often in school did you score 100 percent?

The system is oriented to forcing retailers to fail and it does this by being utterly insensitive to risk, which is surprising because the financial services industry runs on risk management. So if big finance runs on risk management, why are retail payment security rules running away from it?

At the recent Electronic Transactions Association (ETA) conference, Visa and MasterCard executives again defended the standard against industry and government criticisms that the standard is insufficient to prevent breaches. They cited instances where merchants (e.g., Hannaford) and processors (e.g., Heartland) who claimed to be PCI compliant were actually not compliant during the extended period of time during which the breach occurred. That happened because, from a technical perspective, “companies can fall in and out of compliance,” and because sometimes “the PCI assessment is not comprehensive enough,” according to the executives. From these statements, it sounds like the blame for why a “compliant” company can get breached is due to either technology or the assessors. But I maintain that the culprit in all of this is the grading system used to measure PCI compliance, and it’s time for a change. Here’s some arguments in favor of this “modest proposal”:

  • You May Never be Fully Compliant
    PCI Assessments — whether self-assessments or assessments by QSAs — are generally regarded as being valid only for a point in time. But when is that point in time? Is it the day the ROC or SAQ is signed by the assessor merchant, or the day the assessment has been signed off on by the acquirer, or the day that the ROC or SAQ is reviewed and approved by the card network(s)?

    How does a retailer know when that point in time ends? The technical complexity of the controls is inconsistent with the grading system that requires 100 percent to be compliant. Great standard. Bad grading system.

    Thanks to the grading system, and the fact that many of the PCI controls are “volatile” and can be made ineffective by simple configuration or rule changes, this technically means that an organization may never actually be PCI compliant. That’s because, for a typical Level 1 merchant, an assessment will take more than a month, sometimes several months. Thus, it is very possible that between the time the first controls were tested and the time the last controls were tested, changes were made to the first controls such that they are no longer 100 percent compliant.

    Therefore, the merchant is noncompliant even before the ROC or SAQ has been reviewed. Although it is true that some assessments are better than others, the “condition of compliance” is far more subject to the grading system than to the quality of the assessment.

  • advertisement

    12 Comments | Read PCI’s Grading System Is Failing

    1. Steve Sommers Says:


      From time to time we have disagreements but here I am 100% in agreement with you. In the early days of PCI (actually pre-PCS SSC), I was told something that always sticks in my mind. An ex-mucky-muck from one of the associations said: “The card associations view every breach as a compliance failure.” If you put this single statement under a microscope, many of the problems you described in this article become clearer. The problem is, the card associations still defend this belief.

    2. Russell Brown Says:

      Great assessment – we need more outside viewpoints into the PCI requirements and clarity as to how they should be implemented based on the entity.

    3. PCIjeff Says:

      I think you are completely mistaken on your position that PCI requires 100% grade. What about compensating controls? The PCI standard allows merchants and service providers when they are not 100% compliant to the original control to use a compensating control to achieve compliance. The compensating control worksheet is basically a risk assessment of the requirement in question and justification of how the compensating control mitigates the risk to an acceptable level.

      Also, comments below are incorrect
      Thanks to the grading system, and the fact that many of the PCI controls are “volatile” and can be made ineffective by simple configuration or rule changes, this technically means that an organization may never actually be PCI compliant. That’s because, for a typical Level 1 merchant, an assessment will take more than a month, sometimes several months. Thus, it is very possible that between the time the first controls were tested and the time the last controls were tested, changes were made to the first controls such that they are no longer 100 percent compliant.

      A QSA is required to revalidate all time based requirements before completing the ROC. A final compliant ROC does mean the client is compliant at the time of the signature of the Attestation of Compliance form.

    4. Steve Alan Says:

      David – I agree 100% with the comments presented as well as your assesment of “When Bad Assesments are Good” I perform alternate security compliance audits to the PCI and all too often I see and hear from customers that the PCI assesor only took x days, why is my audit taking 2x days. A GOOD auditor is not there to make friends and make the process “painless”. They are there to collect and measure the facts, no matter how long or painful that process may be. IF a customer is following compliance requirements it will be “painless”, but if they are not then it takes longer and they feel the pain. It is all relative

      Keep up the good reporting

    5. David Taylor Says:

      Jeff, I agree that compensating controls introduce risk awareness into the analysis, but my experience and over 370 hours of interviews with merchants, banks, QSAs and service providers indicates they are not used in a consistent manner. They seem like a “bolt on” to the assessment process – which is pretty much what they were.

      My argument is that risk could be made much more an inherent part of the assessment process, with each of the controls risk-weighted. That is, IMHO, where the prioritization scheme published by the SSC recently is headed. I believe if the prioritization system were extended, based on the feedback of the 500+ members of the SSC, to create risk-weights that could be customized for different types of businesses, that the grading system could be improved, so that it would be less of a source of contention than it is today.

      That’s my main point in all of this.
      Dave Taylor

    6. NM Says:

      PCIDSS 1.2 basically requires useless antivirus scans on Linux (not even mentioning AS/400 or Solaris); replacing it with *useful* malware scanning (not all malware are viruses) only counts as a compensating control. This is simply absurd.

      (Tinfoil hat mode: I’m sure Microsoft must have lobbied hard to get the Unix virus scanning exemption removed after 1.1, it must have cost them a lot of biz)

    7. Steve Sommers Says:

      I thought the original wording specifically excluding UNIX was irresponsible for a security doctrine. All operating systems can be susceptible to viruses and other malware. Ironically, the earliest recorded computer viruses were UNIX predecessors and then migrated to various UNIX flavors. Excluding any operating system conveys a false sense of security.

      Now I’m not saying the current wording is perfect. Other forms of malware like key loggers, packet sniffers and memory sniffers are far more dangerous to compromising cardholder data than the average virus. Maybe the wording should be targeted more toward malware prevention, detection and removal as opposed to virus. To me, all viruses are a form of malware but not malware are viruses.

    8. Bruce Sussman, CPA, CISSA CISSP Says:

      Great comments and a very provocative article !

      But one thing is missing from the discussion: the value and importance of entity level controls,the “tone at the top.” As QSA’s, we should rely on the tone set by executive management and the Board to provide us reasonable assurance that controls are maintained. While the sign off is a point in time, good QSAs who care about their clients’ well being and who want to serve the public interest will consider the “tone at the top. ” If PCI compliance is not “baked in” or otherwise integrated with robust policy statements and employee education, PCI may be viewed as a check box exercise at best. “Tone at the top” has nothing to do with the size of an organization but rather with the unexpressed beliefs senior management has about PCI. The best way to discern management’s true opinions about the long term value of PCI is to look at the quality of policy statements and consider whether issues recur.

    9. NM Says:

      @Steve Sommers: what definition of “virus” are you using? “Worms” are not viruses. A virus is a self-replicating program that propagates by inserting itself into executable files. POCs of Linux viruses exist, they have never been a problem in the wild, for a simple reason called “packet management” AKA RPM or DEB.

      There is malware on Linux. There is no virus problem on Linux. This is a fact. (Go ahead and google “linux virus”; go read the pages: they mostly actually talk about non-virus malware)

      I’m not playing on words. You can implement the best protection against Linux-affecting malware, but it won’t be an “antivirus” per PCI.

      Here’s what we’re gonna do where I work: we’re gonna deploy an antivirus, knowing that it will significantly expand the attack surface, because they have to run as root to be able to scan all the files, for no significant gain. This antivirus will mostly scan for Windows virus, even though we don’t have a single Windows machine in our PCI perimeter (we use a completely isolated network, dual workstations and all). Just to check a box.

      Security gain? None.

      Security loss? The AV vendor could be compromised, allowing an attacker in. More likely, the AV daemon could be used to elevate privileges.

    10. Fourat Says:

      Great article!

      For some reason, everyone is fixated on the actual PCI process rather than the corruption that is behind the scenes.

      The associations recently changed from non-profit organizations to for profit. With that said, and to the point in the article that the merchants are already paying higher fees on card-not-present transactions, ALL the risk is still on the merchant/Acquirer. So why would retailers/eCommerce businesses pay more for transactions if none of this risk is passed on to the association/issuing bank? They are collecting more for these transactions, so shouldn’t some of the risk be passed on to them? Merchants should pay the same for interchange (except maybe for rewards/commercial cards) on all transactions since there really is no risk to the association/issuing bank. Their wasted dollars on the so called “Higher Risk” should be spent on preventing risk instead of playing the Association game. Also, the Acquirer is ultimately liable for any / all compromises, yet none of the padded interchange is passed on to them.

      Isn’t paying more based on risk considered insurance? Not here, it is a gimmick for the Associations to get the issuing banks to push their brand instead of their competitor. This is evident with the lawsuit that Amex filed against V/MC a few years back because they are demanding the banks that issue their brand only issue their brand.


      In most philosophical discussions about payment card security (except when they emanate from visionaries like Steve Mott), what usually gets overlooked is the obvious. Credit cards were invented before the Internet, and were never conceived to exist in a card-not-present environment teeming with crafty and invisible electronic bandits.

      For a two-sided marketplace, our current efforts for minimizing theft and compromise are very one-sided (all responsibility on the merchant, none on the cardholder). It’s like sending your innocent daughter to the store with no clothes on. Then demanding that the storekeeper ensure that she isn’t gawked-at or abused.

      As long as we are stuck with a lop-sided program, meaning until some time when the issuing side takes up some responsibility, PCI DSS is our best bet for at least tossing her a blanket.

    12. Chuck Admirer Says:

      Chuck is almost right. It’s more like sending your attractive, voluptuous, naked, exhibitionist daughter to the store and insisting that it is the merchant’s responsibility to provide blindfolds for all parties and to enforce a no peeking rule. btw – Failure to do so makes you liable to great financial penalties and a public spanking.


    StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

    Most Recent Comments

    Why Did Gonzales Hackers Like European Cards So Much Better?

    I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
    Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
    A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
    The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
    @David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

    Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.