Evan Schuman's StorefrontBacktalk

The whole idea of surprise pen testing comes from a military mindset: the “surprise” inspection (which usually wasn’t that much of a surprise anyway). But no senior officer would launch a surprise inspection if there was any chance that a base would be under actual assault—and E-Commerce sites are always at risk. Surprise inspections were a reasonable model for testing datacenter security in pre-Internet days. Today, that model is all wrong.

That doesn’t mean pen testing can’t work. But it means the pen test has to be coordinated with someone in the CIO’s office who can keep a secret (preferably someone who would like to see Security and Operations people sweating) and can quietly supervise the test—and tell the CIO to call a halt if things get out of hand.

That supervisor would make sure the test window isn’t at a bad time for the business, would know in advance the exact time and scope of the test, and would monitor response to make sure that’s the only attack going on. And forget about getting an E-mail with results days later—that pen-test supervisor should be on the phone with the security company all through the test.

If a planned DDoS pen test suddenly starts getting access to customer information, then either the tester has gone out of scope or another attacker has jumped in. If the pen test is supposed to attack security for one group of servers and a different group starts throwing up alerts, something is very wrong.

That’s exactly the type of thing someone inside the IT organization should be watching for, and it can’t happen if the pen test is a true surprise. And if it’s a true surprise, you’re always at risk for a nightmare scenario: some inexperienced IT operations guy who sees a massive attack in progress decides to play hero, and turns a routine test into a full-scale outage.

Besides, there’s a host of other issues that surprise pen testing drags in today—things that weren’t an issue even a decade ago. Suppose the pen tester gets access to payment-card information. Does that trigger PCI problems? What if it’s personal information of customers? Are there state laws or federal regulations that kick in? Remember, pen testing by its nature mimics what thieves do, and that involves going in unexpected directions—sometimes unexpected by even the testers. You don’t want to turn a routine test into an FTC fine or PCI assessment failure, either.

Again, there’s nothing wrong with keeping pen tests a secret from your security and ops people. Will that make them a little paranoid? Sure, but they should be a little paranoid when it comes to security. But forget about random timing and scope. There’s too much risk for the business.

When the top of the IT food chain doesn’t know what’s going on, there’s no such thing as a nice surprise.