Foursquare Crashes Risk Shaking Retailer Loyalty

Written by Frank Hayes
October 5th, 2010

Check-in services like Foursquare are supposed to help keep customers engaged with retailers. But 3-million-user Foursquare was offline for 11 hours last Monday (Oct. 4) due to a database problem—and the same problem took it down for another six hours the next day. And while those outages were happening, it wasn’t just Foursquare that felt the pain; that also came down on the retailers whose customers use the service.

That’s a key problem with tying your loyalty or rewards programs to an outside company, especially a quickly rising startup with inevitable growing pains. From the point of view of your customers, when they check in with Foursquare at your store or restaurant, they’re checking in with you. And if Foursquare is unresponsive, you’re the one who will get blamed.

The technical problem that took down Foursquare is one that, in theory, shouldn’t have happened. Foursquare uses a document-oriented database called MongoDB that’s designed to be highly scalable in ways that relational databases can’t be. Foursquare counted on the database’s ability to balance the load in storing user check-ins.

But according to the company, at 11 a.m. Monday (New York time), one section of the database—a “shard,” in MongoDB jargon—got overloaded. For 90 minutes, Foursquare techs tried to rebalance the load. Nothing worked. At 12:30 p.m., they tried to manually create a new shard to split the data. That caused the entire site to go down.

After another six hours of futile efforts to fix the problem, followed by five more hours of reindexing and testing, the database was working again, and Foursquare was finally back up and accepting check-ins.

The next day it happened again—the same problem, but this time the service was only down for six hours, from 6:30 p.m. to 12:30 a.m.

That was Foursquare’s problem. Your problem as a retailer, as you head into the holidays, is that your check-in-oriented customers aren’t likely to see a real line dividing Foursquare from you. To them, when Foursquare is down, you’re down.

Maybe that sounds painfully familiar. It should. Only a few years ago, you were struggling to keep your Web site working during the holiday season. And just as you worked all the kinks out of your internal operations, you began to run up against the problems of your partners.

You had outsourced your site’s shopping cart, and that collapsed under too heavy a load. You depended on UPS and FedEx to handle shipping information on your site, and their response time slowed to a crawl when things got too busy. Every time you handed off a function to partners, you faced the possibility that they wouldn’t come through—and no matter how crisp your own operations were, they could make you look bad. It was up to you to find ways of making sure they didn’t do that.

That’s pretty much the state of check-in services today, too. Can you pressure Foursquare and other check-in vendors to improve their performance and reliability? Perhaps not. But you can insist that they participate in the same dry-runs that you insist of your own team. After all, for all practical purposes, they now are.


2 Comments | Read Foursquare Crashes Risk Shaking Retailer Loyalty

  1. Rob Rice Says:

    Not sure I agree that consumers will blame retailers for a Foursquare outage. I was stuck in this blackout Tuesday night, when trying to checkin at 5 different retailer stores (it was a big shopping errands evening for me). My conclusion is that this is the last straw and I don’t intend to use Foursquare anymore – it wasted my time! The straw leading up to it can be blamed on retailers, however. I’ve been tired of retailers NOT investing in “Mayors” of their stores [with promotions]. Sure there are examples of trial promotions, but I let down my guard and lost mayor status at 3 majors retailers in the past month because being mayor did not matter, and cashiers proved it. However I still intend to shop at these stores because they add value to my life in other ways. So I chalk it up to poor execution of a neat business model. Shopkick looks a bit more promising but I haven’t seen it yet in stores I frequent. Time will tell if the extra time to play the checkin game is worth the effort or not.

  2. A reader Says:

    From a risk management point of view, the overall system is working fine. There’s a contract (you do have a contract, don’t you?) and if their system is unavailable the contract specifies remedies, such as discounts on future service or perhaps fines. Economically speaking, everything’s still in balance.

    And when someone asks, “You mean that when their system goes down, our customers can’t use it until some kid over there fixes a database? What’s going to happen to my customers?” nobody cares. There’s a contract, and you got your money. Risk managed.

    But your customers are walking down the street, because now that it’s down they realize the whole “mayor” thing was getting kind of boring. And the pub down the street has that new iPhone app thing that blinks like a rave the closer you get to “party critical mass”, so that’s where they’ll be drinking for the next six months.

    So you didn’t factor the faddish nature of marketing schemes into your risk management plan? You didn’t realize foursquare would have a limited popularity lifetime? Or that when a social network app designed to create parties fails, it’s forever doomed as a buzzkiller?

    It’s obvious now that you should have predicted the future better. According to risk management theory, anyway.


StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

Most Recent Comments

Why Did Gonzales Hackers Like European Cards So Much Better?

I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
@David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.