PCI’s Grading System Is Failing
Written by David TaylorGuestView Columnist David Taylor is the Founder of the PCI Knowledge Base, Research Director of the PCI Alliance and a former E-Commerce and Security analyst with Gartner.
For months, retailers and Congress have been attacking retail security standards, but few realize that the problem is not in the standard itself. The problem is a grading system that causes most retailers to be out of compliance most of the time because the rules require 100 percent compliance. How often in school did you score 100 percent?
The system is oriented to forcing retailers to fail and it does this by being utterly insensitive to risk, which is surprising because the financial services industry runs on risk management. So if big finance runs on risk management, why are retail payment security rules running away from it?
At the recent Electronic Transactions Association (ETA) conference, Visa and MasterCard executives again defended the standard against industry and government criticisms that the standard is insufficient to prevent breaches. They cited instances where merchants (e.g., Hannaford) and processors (e.g., Heartland) who claimed to be PCI compliant were actually not compliant during the extended period of time during which the breach occurred. That happened because, from a technical perspective, “companies can fall in and out of compliance,” and because sometimes “the PCI assessment is not comprehensive enough,” according to the executives. From these statements, it sounds like the blame for why a “compliant” company can get breached is due to either technology or the assessors. But I maintain that the culprit in all of this is the grading system used to measure PCI compliance, and it’s time for a change. Here’s some arguments in favor of this “modest proposal”:
PCI Assessments — whether self-assessments or assessments by QSAs — are generally regarded as being valid only for a point in time. But when is that point in time? Is it the day the ROC or SAQ is signed by the assessor merchant, or the day the assessment has been signed off on by the acquirer, or the day that the ROC or SAQ is reviewed and approved by the card network(s)?
How does a retailer know when that point in time ends? The technical complexity of the controls is inconsistent with the grading system that requires 100 percent to be compliant. Great standard. Bad grading system.
Thanks to the grading system, and the fact that many of the PCI controls are “volatile” and can be made ineffective by simple configuration or rule changes, this technically means that an organization may never actually be PCI compliant. That’s because, for a typical Level 1 merchant, an assessment will take more than a month, sometimes several months. Thus, it is very possible that between the time the first controls were tested and the time the last controls were tested, changes were made to the first controls such that they are no longer 100 percent compliant.
Therefore, the merchant is noncompliant even before the ROC or SAQ has been reviewed. Although it is true that some assessments are better than others, the “condition of compliance” is far more subject to the grading system than to the quality of the assessment.
-Marc
April 30th, 2009 at 12:43 am
David,
From time to time we have disagreements but here I am 100% in agreement with you. In the early days of PCI (actually pre-PCS SSC), I was told something that always sticks in my mind. An ex-mucky-muck from one of the associations said: “The card associations view every breach as a compliance failure.” If you put this single statement under a microscope, many of the problems you described in this article become clearer. The problem is, the card associations still defend this belief.
April 30th, 2009 at 7:45 am
Great assessment – we need more outside viewpoints into the PCI requirements and clarity as to how they should be implemented based on the entity.
April 30th, 2009 at 9:47 am
I think you are completely mistaken on your position that PCI requires 100% grade. What about compensating controls? The PCI standard allows merchants and service providers when they are not 100% compliant to the original control to use a compensating control to achieve compliance. The compensating control worksheet is basically a risk assessment of the requirement in question and justification of how the compensating control mitigates the risk to an acceptable level.
Also, comments below are incorrect
Thanks to the grading system, and the fact that many of the PCI controls are “volatile†and can be made ineffective by simple configuration or rule changes, this technically means that an organization may never actually be PCI compliant. That’s because, for a typical Level 1 merchant, an assessment will take more than a month, sometimes several months. Thus, it is very possible that between the time the first controls were tested and the time the last controls were tested, changes were made to the first controls such that they are no longer 100 percent compliant.
A QSA is required to revalidate all time based requirements before completing the ROC. A final compliant ROC does mean the client is compliant at the time of the signature of the Attestation of Compliance form.
April 30th, 2009 at 9:55 am
David – I agree 100% with the comments presented as well as your assesment of “When Bad Assesments are Good” I perform alternate security compliance audits to the PCI and all too often I see and hear from customers that the PCI assesor only took x days, why is my audit taking 2x days. A GOOD auditor is not there to make friends and make the process “painless”. They are there to collect and measure the facts, no matter how long or painful that process may be. IF a customer is following compliance requirements it will be “painless”, but if they are not then it takes longer and they feel the pain. It is all relative
Keep up the good reporting
April 30th, 2009 at 10:18 am
Jeff, I agree that compensating controls introduce risk awareness into the analysis, but my experience and over 370 hours of interviews with merchants, banks, QSAs and service providers indicates they are not used in a consistent manner. They seem like a “bolt on” to the assessment process – which is pretty much what they were.
My argument is that risk could be made much more an inherent part of the assessment process, with each of the controls risk-weighted. That is, IMHO, where the prioritization scheme published by the SSC recently is headed. I believe if the prioritization system were extended, based on the feedback of the 500+ members of the SSC, to create risk-weights that could be customized for different types of businesses, that the grading system could be improved, so that it would be less of a source of contention than it is today.
That’s my main point in all of this.
Dave Taylor
May 5th, 2009 at 11:04 am
PCIDSS 1.2 basically requires useless antivirus scans on Linux (not even mentioning AS/400 or Solaris); replacing it with *useful* malware scanning (not all malware are viruses) only counts as a compensating control. This is simply absurd.
(Tinfoil hat mode: I’m sure Microsoft must have lobbied hard to get the Unix virus scanning exemption removed after 1.1, it must have cost them a lot of biz)
May 6th, 2009 at 12:18 pm
I thought the original wording specifically excluding UNIX was irresponsible for a security doctrine. All operating systems can be susceptible to viruses and other malware. Ironically, the earliest recorded computer viruses were UNIX predecessors and then migrated to various UNIX flavors. Excluding any operating system conveys a false sense of security.
Now I’m not saying the current wording is perfect. Other forms of malware like key loggers, packet sniffers and memory sniffers are far more dangerous to compromising cardholder data than the average virus. Maybe the wording should be targeted more toward malware prevention, detection and removal as opposed to virus. To me, all viruses are a form of malware but not malware are viruses.
May 6th, 2009 at 3:23 pm
Great comments and a very provocative article !
But one thing is missing from the discussion: the value and importance of entity level controls,the “tone at the top.” As QSA’s, we should rely on the tone set by executive management and the Board to provide us reasonable assurance that controls are maintained. While the sign off is a point in time, good QSAs who care about their clients’ well being and who want to serve the public interest will consider the “tone at the top. ” If PCI compliance is not “baked in” or otherwise integrated with robust policy statements and employee education, PCI may be viewed as a check box exercise at best. “Tone at the top” has nothing to do with the size of an organization but rather with the unexpressed beliefs senior management has about PCI. The best way to discern management’s true opinions about the long term value of PCI is to look at the quality of policy statements and consider whether issues recur.
May 7th, 2009 at 5:17 am
@Steve Sommers: what definition of “virus” are you using? “Worms” are not viruses. A virus is a self-replicating program that propagates by inserting itself into executable files. POCs of Linux viruses exist, they have never been a problem in the wild, for a simple reason called “packet management” AKA RPM or DEB.
There is malware on Linux. There is no virus problem on Linux. This is a fact. (Go ahead and google “linux virus”; go read the pages: they mostly actually talk about non-virus malware)
I’m not playing on words. You can implement the best protection against Linux-affecting malware, but it won’t be an “antivirus” per PCI.
Here’s what we’re gonna do where I work: we’re gonna deploy an antivirus, knowing that it will significantly expand the attack surface, because they have to run as root to be able to scan all the files, for no significant gain. This antivirus will mostly scan for Windows virus, even though we don’t have a single Windows machine in our PCI perimeter (we use a completely isolated network, dual workstations and all). Just to check a box.
Security gain? None.
Security loss? The AV vendor could be compromised, allowing an attacker in. More likely, the AV daemon could be used to elevate privileges.
May 7th, 2009 at 11:20 am
Great article!
For some reason, everyone is fixated on the actual PCI process rather than the corruption that is behind the scenes.
The associations recently changed from non-profit organizations to for profit. With that said, and to the point in the article that the merchants are already paying higher fees on card-not-present transactions, ALL the risk is still on the merchant/Acquirer. So why would retailers/eCommerce businesses pay more for transactions if none of this risk is passed on to the association/issuing bank? They are collecting more for these transactions, so shouldn’t some of the risk be passed on to them? Merchants should pay the same for interchange (except maybe for rewards/commercial cards) on all transactions since there really is no risk to the association/issuing bank. Their wasted dollars on the so called “Higher Risk†should be spent on preventing risk instead of playing the Association game. Also, the Acquirer is ultimately liable for any / all compromises, yet none of the padded interchange is passed on to them.
Isn’t paying more based on risk considered insurance? Not here, it is a gimmick for the Associations to get the issuing banks to push their brand instead of their competitor. This is evident with the lawsuit that Amex filed against V/MC a few years back because they are demanding the banks that issue their brand only issue their brand.
May 8th, 2009 at 3:16 pm
In most philosophical discussions about payment card security (except when they emanate from visionaries like Steve Mott), what usually gets overlooked is the obvious. Credit cards were invented before the Internet, and were never conceived to exist in a card-not-present environment teeming with crafty and invisible electronic bandits.
For a two-sided marketplace, our current efforts for minimizing theft and compromise are very one-sided (all responsibility on the merchant, none on the cardholder). It’s like sending your innocent daughter to the store with no clothes on. Then demanding that the storekeeper ensure that she isn’t gawked-at or abused.
As long as we are stuck with a lop-sided program, meaning until some time when the issuing side takes up some responsibility, PCI DSS is our best bet for at least tossing her a blanket.
May 14th, 2009 at 8:31 pm
Chuck is almost right. It’s more like sending your attractive, voluptuous, naked, exhibitionist daughter to the store and insisting that it is the merchant’s responsibility to provide blindfolds for all parties and to enforce a no peeking rule. btw – Failure to do so makes you liable to great financial penalties and a public spanking.