Foursquare Crashes Risk Shaking Retailer Loyalty
Written by Frank HayesCheck-in services like Foursquare are supposed to help keep customers engaged with retailers. But 3-million-user Foursquare was offline for 11 hours last Monday (Oct. 4) due to a database problem—and the same problem took it down for another six hours the next day. And while those outages were happening, it wasn’t just Foursquare that felt the pain; that also came down on the retailers whose customers use the service.
That’s a key problem with tying your loyalty or rewards programs to an outside company, especially a quickly rising startup with inevitable growing pains. From the point of view of your customers, when they check in with Foursquare at your store or restaurant, they’re checking in with you. And if Foursquare is unresponsive, you’re the one who will get blamed.
The technical problem that took down Foursquare is one that, in theory, shouldn’t have happened. Foursquare uses a document-oriented database called MongoDB that’s designed to be highly scalable in ways that relational databases can’t be. Foursquare counted on the database’s ability to balance the load in storing user check-ins.
But according to the company, at 11 a.m. Monday (New York time), one section of the database—a “shard,” in MongoDB jargon—got overloaded. For 90 minutes, Foursquare techs tried to rebalance the load. Nothing worked. At 12:30 p.m., they tried to manually create a new shard to split the data. That caused the entire site to go down.
After another six hours of futile efforts to fix the problem, followed by five more hours of reindexing and testing, the database was working again, and Foursquare was finally back up and accepting check-ins.
The next day it happened again—the same problem, but this time the service was only down for six hours, from 6:30 p.m. to 12:30 a.m.
That was Foursquare’s problem. Your problem as a retailer, as you head into the holidays, is that your check-in-oriented customers aren’t likely to see a real line dividing Foursquare from you. To them, when Foursquare is down, you’re down.
Maybe that sounds painfully familiar. It should. Only a few years ago, you were struggling to keep your Web site working during the holiday season. And just as you worked all the kinks out of your internal operations, you began to run up against the problems of your partners.
You had outsourced your site’s shopping cart, and that collapsed under too heavy a load. You depended on UPS and FedEx to handle shipping information on your site, and their response time slowed to a crawl when things got too busy. Every time you handed off a function to partners, you faced the possibility that they wouldn’t come through—and no matter how crisp your own operations were, they could make you look bad. It was up to you to find ways of making sure they didn’t do that.
That’s pretty much the state of check-in services today, too. Can you pressure Foursquare and other check-in vendors to improve their performance and reliability? Perhaps not. But you can insist that they participate in the same dry-runs that you insist of your own team. After all, for all practical purposes, they now are.
-Marc
October 7th, 2010 at 8:49 am
Not sure I agree that consumers will blame retailers for a Foursquare outage. I was stuck in this blackout Tuesday night, when trying to checkin at 5 different retailer stores (it was a big shopping errands evening for me). My conclusion is that this is the last straw and I don’t intend to use Foursquare anymore – it wasted my time! The straw leading up to it can be blamed on retailers, however. I’ve been tired of retailers NOT investing in “Mayors” of their stores [with promotions]. Sure there are examples of trial promotions, but I let down my guard and lost mayor status at 3 majors retailers in the past month because being mayor did not matter, and cashiers proved it. However I still intend to shop at these stores because they add value to my life in other ways. So I chalk it up to poor execution of a neat business model. Shopkick looks a bit more promising but I haven’t seen it yet in stores I frequent. Time will tell if the extra time to play the checkin game is worth the effort or not.
October 7th, 2010 at 9:13 am
From a risk management point of view, the overall system is working fine. There’s a contract (you do have a contract, don’t you?) and if their system is unavailable the contract specifies remedies, such as discounts on future service or perhaps fines. Economically speaking, everything’s still in balance.
And when someone asks, “You mean that when their system goes down, our customers can’t use it until some kid over there fixes a database? What’s going to happen to my customers?” nobody cares. There’s a contract, and you got your money. Risk managed.
But your customers are walking down the street, because now that it’s down they realize the whole “mayor” thing was getting kind of boring. And the pub down the street has that new iPhone app thing that blinks like a rave the closer you get to “party critical mass”, so that’s where they’ll be drinking for the next six months.
So you didn’t factor the faddish nature of marketing schemes into your risk management plan? You didn’t realize foursquare would have a limited popularity lifetime? Or that when a social network app designed to create parties fails, it’s forever doomed as a buzzkiller?
It’s obvious now that you should have predicted the future better. According to risk management theory, anyway.