This is page 2 of:
Down For 8 Days: American Eagle’s Site Disaster
Enough speculation and job postings. Let’s look at the timeline. On Monday (July 19) in the early evening (New York time), the site fully crashed, according to Jani Strand, the chain’s VP/corporate communications. She said it was “a hardware issue with the host of the site,” which was—and presumably still is—IBM. “IBM is our partner and we’re working with them to solve this. We’re both very disappointed.”
When asked about details of what brought the site down, Strand said “we’re not going into that level of detail.”
After the site crashed on Monday, a screen message told visitors: “Sorry. We need a few minutes to re-organize our closet. We promise to be back in a bit with even more” and then added the logos of the chain’s three key brands: American Eagle Outfitters, Aerie and 77kids.
(Given that the site wouldn’t be fully back up for eight days, the site must use the same definition for “a few minutes” that my 12-year-old daughter uses when she offers a time estimate for when she’ll wash the dishes.)
That sign was up through Wednesday (July 21). On Thursday (July 22), the sign changed to “We’re making updates to our sites. Free Shipping on us when we’re back, thru July 25.” Note: The site didn’t fully return until July 27.
Back to the timeline, the site came back up late Thursday morning (July 22): Strand put the time at “about 11:15 AM” and added that the site at that time had “a little bit of limited functionality. Some of the saved data hasn’t been restored, but it is shoppable now.”
By Friday morning (July 23), the site’s message had changed again, offering some details on the limited functionality. “We’re still working through some issues, but you’re able to shop! Everything should be completely fixed very soon. Thanks for hanging in there. Stuff we’re still working on: Order tracking, Registered Information Functionality, Wish List, Order History.”
Screens through Tuesday (July 27) thanked consumers for “hanging in there while we work through some site issues.” On Tuesday afternoon, the warning messages came down and IBM said the site “is now fully operational.”
Although this outage was much worse than retailers typically suffer through, even a several-hour outage poses the potential for losing customers. As such, does it make sense to look at your mobile site as an emergency backup to your Web site?
A few issues need to be considered before you make that leap: Many mobile sites leverage the content and the database of the main site. That would mean a mobile site would only be helpful if the main databases of the site—pricing, inventory, order placement, etc.—are still functioning. Payment is often independent, so there’s a fine chance that system may survive. If it’s merely a hosting server that has crashed, the mobile site could be a powerful backup.
In American Eagle’s case, this approach would not have worked because of the infrastructure they chose. But if the idea of using mobile as a backup appeals to an IT team, then this should be added to the criteria of a mobile vendor, platform and architecture choice.
The American Eagle Usablenet site was designed as a proxy between the mobile device and the American Eagle Web site, which made mobile backup impossible. “The search page on the mobile device can be served by Usablenet only if the search page on the main web site is working,” said one official who was both involved in the arrangement and is also fond of her anonymity.
Usablenet and other proxy services would not support a mobile failover strategy, but some of the other major mobile outfits would. “If (American Eagle) was using a host like Digby, which is architected to run an application framework independent from the company’s main site, and relies only on an inventory data feed and some backend web services, then they could have operated the mobile site autonomously albeit with some limited functionality,” said the source, who works in IT and is not involved with Digby.
“The underlying message is that if autonomy from mobile to main site is a need, the former should be architected for that purpose and, if a hosting provider is used, then it is critical to choose a host that utilized a framework that make this possible. Some do, but Usablenet by definition will not,” said the IT person.
Before deciding architecture, a company has to decide if using mobile as a Web site backup is even an idea that works for them. This approach might require a mobile site to house duplicate mirrored versions of the key databases, explicitly so that that site could more likely survive a main Web site outage?
Even if it does survive, there are other concerns. Despite ongoing signs that mobile is growing with staggering speed, some mobile sites are not given the infrastructure to support enterprise-level Web traffic. Unless the sites are redesigned to handle much more bandwidth, pointing your Web traffic to your existing mobile site may cause a mobile crash.
Another concern, albeit much less significant, is design. Mobile sites are deliberately kept utilitarian. That look may not appeal to some marketers, who could argue that “no site” is better than a “bland site.” Perhaps, but we’re guessing the CFO may disagree.
July 29th, 2010 at 4:26 am
Contingency planning is frought with all sorts of pitfalls. The suggestion about running your mobile site on “mirrored versions of the key databases” sounds great, aprt from in AE’s case the gradual curruption of the main site’s databases due to the array problem would also be “mirrored” onto the mobile site.
You could handle bandwidth issues by locating in the same datacentre and sharing the main site’s bandwidth. But that leaves both sites vulnerable to both a bandwidth outage or a datacentre failure (say, the power supply fails.
It reminds me of the phrase currently very popular with politians (certainly over here in the UK) “it’s a problem of unintended consequences”.