"Damn, that rock is heavy."
-- Grrrg, son of Booog, discoverer of The Big Data Problem
Big Data is big business, and all the
nerds are talking about it. Since many of my readers are not nerds (and I mean that in the nicest way), I'd like to break down what Big Data is, and then I'd like to put it in a historical context that I think has been somewhat neglected.
People and devices are more connected than ever before. Fortunately for you, you represent only one of those people, and you have only a handful of devices and a few friends. All of your data, and all of the relevant data from the people you care about, fits in less than a cubic foot on your desk. If you search for a particular piece of information stored there, your computer can find it in a couple of seconds. Congratulations; you're doing fine: you don't have a Big Data Problem.
Your ISP has tens of thousands of customers just like you.
LinkedIn had 100 million members as of a year ago.
Facebook has almost 900 million members. And
They.
Save.
Everything. In other words, they have 5-10 orders of magnitude more data than you do. That's far too much to store in one place, and if they searched their data the way you search yours, your one-second search would take them a week. If you typed a query into Google.com and had to wait a week for your search results to come back, you'd be pissed -- like,
really pissed.
That,
my friends, is a Big Data Problem.