Evan Schuman's StorefrontBacktalk

The CIO for Tulsa, Okla., was put on administrative leave on October 1, after a security company hired by the city ran an unannounced penetration test, and no one in the IT department realized it was a test. The usual tut-tutting aside (“How could he forget he hired this outfit?”), we’re wondering whether it’s time to dump the security “best practice” of doing surprise pen tests.

Yes, those tests should be a surprise to the security and ops people. But to the CIO? In today’s legal environment, with PCI and personal information on the line? That’s crazy. For a retailer, it’s even crazier.

What actually happened seems pretty clear at this point: The city’s Web sites were taken offline on September 12, after one of the city’s servers “was targeted by an unknown source,” the city said. Six days later, the city sent out 90,000 letters to people whose personal information was on the server because they had applied for jobs with the city over the past decade. Total cost of the mailing: $20,000. (No, we’re not sure how they sent out 90,000 time-critical, individualized letters for 22 cents each.)

Further investigation turned up an E-mail from the pen-testing company, SecurityMetrics, that was sent a few days after the “breach” with results of the test. Oops. That’s when CIO Tom Golliver was put on leave. The city also spent another $25,000 on a security postmortem, which exposed problems with the police department’s Web site so bad that the Web site is being completely rebuilt.

The standard wisdom on this type of incident is that the breach response checklist should have included checking with the security vendor to make sure it wasn’t the source of the attack. One big problem with that: Just because there’s a pen test going on doesn’t mean you’re not also being attacked by cyberthieves.

After all, that was the tactic used in last year’s huge attack on Sony’s PlayStation Network E-Commerce site: A denial-of-service attack was used as a diversion while thieves targeted information on millions of customers. In that case, both attacks were by the same thieves. But thieves are opportunistic, and they monitor high-profile Web sites constantly. If they spot a site that’s under DoS attack—for real or as a test—that’s the perfect time to launch their own intrusion.

But that’s not the only problem with surprise pen-test attacks, especially for retail chains. There are definitely wrong times for a retailer to get a surprise attack: during a major online sale or promotion, for example, or while newly upgraded equipment is coming online after a planned outage. The point of pen testing is to identify ongoing vulnerabilities—not to interfere with business (or with getting back to business). If the test isn’t coordinated with IT executives, there’s no way to minimize disruption to the business.