Everyone seems to be hyping Big Data right now. I think we’ve reached that slightly scary point where CEOs are aware of Big Data and are beginning to think it a panacea for all business ills. But I’m asking the question: what’s the big deal with Big Data?
We’ve always had more data available to us than we can deal with. Our ability to manage that volume of data has increased over time, although so also has the amount of data we generate. We’re always going to be playing catch-up. But are we any better at turning all this data into useful and valuable meaning?
Very few people seem to be really thinking about the point of Big Data
I attended a recent CloudCamp in January focusing on Big Data and made a few observations:
- techies love to indulge their hoarding tendencies (Must. Store. More. Data.);
- armchair lawyers still love to digress on high-impact, low-probability concerns such as the Patriot Act and how likely cloud tech companies were to be raided by the Feds (although this did provoke an amusing slap-down by Werner Vogels, CTO of Amazon); and
- very few people seem to be really thinking about the ultimate point of being able to store and query such large amounts of data, i.e. what questions will all this data help us to answer?
While being able to store more data and query it more quickly is fundamental to our ability to start deriving meaning from it, I came away from CloudCamp’s Big Data event with the overriding impression that technology vendors in this space are continuing to focus primarily on size and speed (features) rather than facilitating knowledge and insight (benefits).
The problem with this approach is that corporate buyers will likely see the value not so much in Big Data itself, but in the answers to business questions offered by analysis of the Big Data. Organisations want easier solutions to the day-to-day business problems they have such as:
- Where should I build my next retail store?
- How can I identify the people most likely to want to consume my products and services?
- How can I understand more about my current and prospective customers so that I can tailor my content and services to them?
Although the ability to answer these questions absolutely depends on being able to store and query the underlying data, the typical CEO (think: glorified sales person) is not going to appreciate or concern themselves with the capabilities of Hadoop, Hive, noSQL or any other Big Data technology. In other words, they don’t care how you get to the answer – they just want the answer. We know Einstein’s equation E=mc2, but history is more concerned with its meaning rather the type of paper on which he wrote the proof.
So where does this lead us? Transactional information is becoming less dense in meaning so we need to be able to store and query larger sets of data before we can start to see the underlying picture. Big Data provides us with the ability to do this, which in turn allows us to find and articulate meaning within that data.
What I feel we’re lacking are similarly strong analytics tools, designed to cope with this volume of low-density information. In other words, we can store all this information pretty well, but we can’t unlock the meaning of that information quite as easily.
Big Data is a big deal. But our challenge remains to ask the right questions, understand the answers hiding in our data and then turn those answers into the valuable insight that will then drive those business decisions. Big Data is a great foundation, but I’m much more interested in seeing how Big Analytics turns out.
Are you a product professional working with Big Data? Do you agree or think I’m completely missing the point? Let me know in the comments.