Living the Big Data Dream: Confidence, Confidentiality and Continuous Automation in the 21st Century

Big Data Evangelist, IBM

Big data is about much more than scaling, accelerating and broadening your analytic applications. Without trustworthy data to fuel it all, you can’t have much confidence in the hidden patterns that big data reveals, the decisions it drives, or the outcomes that you may or may not achieve from it all.

Confidence is a fragile thing, as Richard Lee pointed out in this recent blog. The big data dream rides in great part on the veracity of the data, and that, in turn, depends on strong data governance, lineage traceability and robust data integration automation. Lee discusses the very real risk of organizations falling into a big-data “crisis of confidence” if they don’t institute tight end-to-end governance processes, controls and failsafes for assuring the continuous veracity of their big-data analytics assets. David Corrigan’s recent blog discusses the need for governance “zones” that align with the different sources and use cases of big data. And also see my recent blog on the layered technical approach you need to institute to ensure continuous data and decision confidence across all of your business analytics initiatives, not just big data.

Bear in mind that confidence is not only about the veracity of your data. The confidentiality, security and privacy of your big data assets are also of critical concern. The continued confidence of all your stakeholders—customers, employees, partners, investors, regulators and so forth—depends on your ability to keep your business data securely out of the reach of unauthorized parties. In addition, your customers need to have confidence that your powerful big-data infrastructure won’t invade their privacy as they, for example, use smartphones and other new mobile and “Internet of Things” (IoT) gadgets to engage with you. See this recent LinkedIn post by me on some of the privacy sensitivities surrounding mobile/IoT adoption in business-to-consumer applications. And see this LinkedIn post on the privacy sensitivities that data scientists will run into as they build more powerful analytic models from data at any scale.

In addition to veracity and confidentiality issues, confidence in big data’s potential also rests on the unprecedented extent to which automated programs can help organizations to identify data-driven insights nonstop. But this very same degree of automation also unnerves some people, who may regard big data as an agile machine for encroaching on every aspect of their lives. Recently, I’ve posted many thoughts to LinkedIn on the potential and pitfalls of continuous big-data integration, including:

As history has shown, economic conditions improve or deteriorate based on the intangible power of popular confidence. In the 21st century, we now realize that big data—in all of its manifestations—is a huge component of the confidence equation.

To a growing degree, popular confidence in all private and public institutions rides on our ability to manage big-data operations responsibly.