How Long Is A Point-In-Time Audit Good For?

Written by David Taylor
January 28th, 2009

GuestView Columnist David Taylor is the Founder of the PCI Knowledge Base, Research Director of the PCI Alliance and a former E-Commerce and Security analyst with Gartner.

Heartland Payment Systems had a large security breach that apparently started in spring 2008. Around the same time, Trustwave–the largest of the QSA assessors–affirmed the company as PCI compliant. All sorts of things can be read into this finding: There was a problem with the audit; there is a problem with the PCI standards; or there’s a big difference between being PCI compliant and preventing security breaches.

In the last week, I have spoken with quite a few retailers and service providers who would like to condemn the whole process. After all, they argue, wasn’t the original purpose of spending all this money on security to prevent breaches like Heartland (or TJX or a thousand others)?

  • Compliance Is Not Security
    Every security professional worthy of their myriad certifications knows that security and compliance are two different animals. Security is about minimizing risk. It’s never stated as an absolute. Compliance, on the other hand, is about minimizing liability. It’s almost always stated as an absolute.

    I’m deliberately making this distinction simply to point out that, even though security professionals “get this,” many business executives have been “sold” on the idea of spending money for compliance as a way to stop security breaches. So if a company that is “compliant” has a security breach, then its security and compliance managers will get frantic phone calls from business executives who want to be reassured that their spending on compliance has not been in vain. There may even be some yelling.

  • How Long Should Point-In-Time Last?
    All PCI QSAs worthy of their certifications will tell you that their assessment is a “point-in-time” audit. After all, with 200+ controls to review, how could it be anything else? But how long is a “point in time”? And is there any way to make that point in time last longer, so that a “state of compliance” can persist for months–or at least until the next “point-in-time” review?

    At least two points are worth making here. First, a PCI assessment (whether by a QSA or self assessment) takes time, often several months. It is common for some controls to have been reviewed two or three months before the final report on compliance (ROC) is submitted. And another month or two may pass before that ROC is reviewed and accepted. So it is not unreasonable to believe that by the time a ROC is finally approved, some of the more “dynamic” controls (e.g., firewall rules, system patches and configuration, identity management) may have already changed to the extent that they are no longer in compliance. And the implication is that in some cases a company may never be fully compliant.

    Second, we have often commented about the “bi-polar” nature of the compliance process. Project managers run around like crazy ramping up to a PCI audit, gathering data, setting up interviews with assessors, compiling reports. And then they (and their security budgets) are exhausted once they get the “green ROC,” or they move on to the next project. As a result, we maintain that any controls that can be “automated” should be automated. Control effectiveness data should be collected on a continuous basis. That is the best currently available option for prolonging a state of compliance. The company then will be able to provide evidence that control effectiveness did not “erode” after the audit was completed.

  • The Use Of Absolutes And Sampling
    Over its various versions, the language of the PCI security standards has evolved to use fewer “absolutes” (never, always, all, must) in the review of controls. Some remain, of course, and they will continue to be a reason why a company can easily fall out of compliance after a point-in-time audit. Another issue, less frequently discussed, is sampling. There are more than 20 mentions of sampling in the PCI assessment procedures. The goal, in each case, is to ensure that the sample is “representative” of the overall population of systems. It’s a safe bet, however, that few QSAs have sufficient information to understand the overall population of the systems they are sampling.

    QSAs must be guided by their customer. But many PCI project managers and the technicians who are interviewed for the audit may not, themselves, understand enough about the population of systems to be sampled. My point here is simple: After a breach, the forensic process is likely to review the entire population of systems, not just a sample. Thus, it is easy to understand how a sampling process could miss vulnerabilities and even malware, especially if the “bad people” were smart enough to hide the malware in some “out of the way” place, such as an unallocated section of a disk drive or in temp files, as they did at Heartland. Thus, I would argue that it is possible for a completely correct audit to miss certain types of “passive” malware, such as sniffers, if the criminals are especially clever.

  • Are These Things Flaws In PCI?
    I’m not saying that PCI is flawed because it’s based on sampling, requires a 100 percent score to pass or takes months to complete. Nor is it because the company being audited could actually have fallen out of compliance by the time an audit is complete. I am saying that security is not perfect; nor is the audit process. Please note that I’ve avoided saying “PCI” in most cases because my comments apply to other types of audits as well.

    Rather, the whole point here is to lobby for more awareness of the “variability” of the process. If there is any one change I would like to see, it is more incorporation of measures of “control effectiveness”–into the standard, the assessment process and the automation of controls reporting, and even in how people think of compliance. If the definition and metrics associated with compliance were adjusted to more accurately mirror the measurement of risk and security (both highly variable), we could more accurately set both the expectations and the budgets for IT security and compliance.

  • The Bottom Line
    Our team at the PCI Knowledge Base has established our research agenda for 2009, working with the National Retail Federation and, potentially, a couple of other industry groups. We will be focusing more on PA-DSS and the PCI PED TDES requirements and helping organizations identify solution providers who can deliver the PCI Best Practices that our 2008 research program has identified. We will be publishing a report shortly on the 2008 PCI Best Practices research. If you want more information, just send an E-mail to

  • advertisement

    Comments are closed.


    StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

    Most Recent Comments

    Why Did Gonzales Hackers Like European Cards So Much Better?

    I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
    Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
    A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
    The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
    @David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

    Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.