Big Data Is Exactly What You Think It Isn’t

Written by Todd L. Michaud
May 16th, 2012

Todd Michaud runs Power Thinking Media, which helps retailers and restaurants tackle the convergence of social, mobile and retail technologies. He spent nine years delivering technology solutions to more than 10,000 retail locations as VP of IT for Focus and Director of Retail Technology for Dunkin’ Brands.

Who cares about Big Data? You should. All of a sudden, Web logs that were kept simply for troubleshooting purposes can now be mined to determine valuable information about customers’ preferences.

Logs that are created by physical machines can now be analyzed en masse to look for information to help advance a business. Data from social networks can now be mined for customer sentiment. These problems were too big and too complex before. But now, answers are within reach.

The Big Data space is filled with so many posers, fakers and wannabes it’s ridiculous. Everybody is trying to catch the Big Data wave by getting their name attached to this hot new trend. Let’s start by all getting on the same page in terms of what I consider the definition of Big Data:

When someone in an organization gets the idea that they would like to pull useful information out of bunch of data that is so large and complex that the CIO says, “Well, how the hell are we going to do that?” That is Big Data.

What’s new is that a bunch of really smart people have solved the two biggest challenges when it comes to analyzing data of large size and complexity. First, they have removed the need for giant, expensive, specialized hardware platforms (instead using a large number of small “commodity servers” or even cloud servers). And second, they have also removed the need to structure the data in a given format prior to running analysis.

These two technological advancements (and the dozens of other underlying technologies that support them) have unleashed a tremendous number of possibilities when it comes to gaining insight from information that would have previously required millions of dollars worth of hardware and a staff of data experts to process. In fact, with open-source software and Amazon EC2 virtual hardware, you can now process a job on 1,000 servers that, even if it lasted for two hours, would cost less than $200.

Being in the Big Data space, I try to read as much as I can that is published on the topic, and I find myself pleasantly surprised when I finish an article that doesn’t leave me wanting to get those 3 to 5 minutes of my life back.

Unlike the “cloud” hype-cycle, where every company on the planet decided to start pitching the fact that it is, in fact, “in the cloud” (including helping the poor housewife catch her missed episode of TV), this insanity seems to be brought upon by writers and analysts who are covering the space but do not have the slightest idea what they are writing about. Cramming buzzwords into a story may help with search engine rank, but it sure does give me a headache.

Regardless of the software platform, the database and the data analysis components that may win or lose in this land grab, I see Big Data going through a three-phase evolution:

  • Decisions Support. The first phase of Big Data will be as companies implement systems that create dashboards and reports that help them run their businesses. Big Data insights will help executives determine how to best steer their companies. This might be in the form of determining new product launches, existing product enhancements, potential market opportunities, etc.


One Comment | Read Big Data Is Exactly What You Think It Isn’t

  1. Thad Peterson Says:

    Todd- Great article, thanks for clearing the “clouds” away from the “big data” hoopla. I agree that it’s going to completely change the way retailers deal with their customers (and their suppliers), but there’s going to be a whole bunch o’ garbage out there before people really understand. Thanks for helping.


StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

Most Recent Comments

Why Did Gonzales Hackers Like European Cards So Much Better?

I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
@David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.