Database Corruption Blamed For Netflix Snafu

Written by Evan Schuman
September 3rd, 2008

The IT chief at Netflix has pointed the finger of blame for its site problems last month at "a database corruption event in our shipping system." The problem prevented customers from receiving their DVDs for about three days.

Mike Osier, head of IT Operations at Netflix, wrote in a blog posting on Aug. 22 that it was "a key faulty hardware component" that caused the hiccup. "On Monday, 8/11, our monitors flagged a database corruption event in our shipping system," Osier said. "Over the course of the day, we began experiencing similar problems in peripheral databases until our shipping system went down. It was going to be a long night."

It wasn’t business as usual for the company until the end of that week, when on Aug. 15 Netflix began shipping from all of 55 distribution centers again and three million DVDs were mailed out.

"We suspected hardware and moved the shipping system to an isolated environment, gradually getting DVD shipments moving again. Eventually the system was repaired and shipping returned to normal conditions," Osier said. "With some great forensic help from our vendors, root cause was identified as a key faulty hardware component. It definitively caused the problem yet reported no detectable errors. We’ve taken steps to fortify our shipping system with the acquisition of additional equipment and worked with our vendors to verify we’re in good shape elsewhere."

One sidenote: Netflix gets a healthy kudos for having come clean—theoretically—with what was behind the glitch, a move that is quite rare today with retail problems. The comments were posted on Netflix’s blog: The problem? Go to and try and find a link—any link—to the blog. Not on the homepage, not in the About Us section, not in the Contact Us section.

If an E-Commerce site goes to all of the trouble of creating a blog and filling it with seemingly good content, why not link to it—and link to it prominently—from the site?


Comments are closed.


StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

Most Recent Comments

Why Did Gonzales Hackers Like European Cards So Much Better?

I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
@David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.