Big data means more data – from more sources, in more formats. It also involves data that is created at a more rapid pace. All of those factors make it harder to establish context. Where did this data come from? How much do you trust it? What steps were taken to correct, or massage, the data?
In the race to exploit big data for competitive advantage, companies may be making out-of-context decisions: sub-optimal or even incorrect decisions. Big data is complex – there are many data types and formats to utilize in combination to get the best insights. Often the complexity means you may factor in only some of the data you need. You’ll get an answer, but did you have the full context to really make the best decision?
When surveyed in a recent Forrester research paper, nearly 50% of executives feel their workers always have the right context (i.e., information) to make good decisions; a further 45% think they have the right context “in some cases.” How do workers feel? Well, only 25% think they have the right context, and 70% replied “in some cases.”
That’s a big gap. Unfortunately “in some cases” typically means “sort of,” which you can also translate to “not so much.” And our appetite for big data is making this problem even worse.
The problem of context becomes even more pressing when you think from the point of view of a use case. Forget about context around the data that you have, what context do you need for your use case? What data do you need to make the best possible decision? Now, where does that data come from and what steps need to be taken to improve your confidence in that data. Context must be thought of from the POV of a use case – first ask what you are doing, then ask what data you need, and how confident you need to be in that data to act.
Context is one of the sleeper issues in the big data market, and the market is waking up to address this important issue. You simply can’t dump all data into a new big data system and confidently get accurate results. You need context. Context of where the data came from, what happened to it along the way, whether it was governed appropriately, and if you are using all of the available data to make the right decisions. Context is metadata – data on big data that documents where it came from, how it was governed and improved, a business glossary of terms and definitions for the data, the confidence level in that data, and what it should be used for. More and more organizations are implementing metadata before beginning with their big data projects, essentially creating a big data catalogue from which they can browse, find, and use big data in context.
Successful big data & analytics projects result in actions. Actions are only taken when you’re confident. And confidence stems in large part from having proper context. Contest, therefore, is crucial to utilizing big data successfully.
Forrester Research published a very telling report – Big Data Needs Agile Information Integration & Governance, which delves into this issue of context in more detail.