Deriving Innovation from a Data-Driven Mind-set: Part 1

Apply big data analytics to inspire fresh ways to identify, frame, and resolve business problems

An organization’s capability to frequently innovate is crucial for business growth and often necessary for survival. Leaders of fast-paced global businesses in an uncertain industry environment need to continuously seek innovation that can revitalize business models and processes that may have lapsed into rigid existences. However, they are also aware that innovation can be challenging and fraught with uncertainty.

Big data analytics, in addition to many other business benefits, can guide the innovation process to enhance its efficiency, effectiveness, and predictability. Big data analytics promotes a data-driven mind-set that listens to the data for new insights and disrupts entrenched thinking that may hinder innovation. In addition, it triggers and supports what-if analyses to help predict the impact of new ideas on key business metrics and then uses evidence-based business performance analysis to track their implementation.

Integrating big data analytics into business planning and operational processes can provide valuable feedback loops and enable an adaptive innovation process. In short, big data analytics helps to spark innovation, guide its refinement and adoption, and track its ongoing benefits.

The two sides of big data analytics

Big data has been credited with everything from improving business performance to contributing to political campaigns, sporting victories, and medical breakthroughs. Such hype begs the question of whether it’s just big talk or if there is real value to it. Certainly, there is no shortage of very credible success stories demonstrating how various entities have derived significant value from big data analytics.

Editor's note: Part 3 of this article series presents several case studies that demonstrate a range of big data successes and innovations.

The term big data can be commonly used to simply stress how big and how fast data is currently being generated. This description is of course true, but data—whether its volume is big or small—does not do anything by itself. It has to be manipulated, interrogated, and mined for insights to be of any value to an organization. In other words, data’s value is obtained through analyzing it. Big data analytics ties big data (the passive side) to analytics (its active side).

Data has long been generated in larger volumes than can be consumed, and much of it is often wasted. Now, it is increasing in an exponential manner; just 30 years ago, the IT industry talked in terms of megabytes for storage high-end capacity, and now it speaks in terms of terabytes—a one million times increase. Speed and capacity of processing have also increased correspondingly. This rise is fundamental to big data analytics because as Victor Mayer-Schönberger and Kenneth Cukier said in their 2013 book, “The change in the scale led to a change of state. The quantitative change has led to a qualitative one.”1

Tools such as Google Map Reduce and its open source equivalent Apache Hadoop were invented to corral these massive data sets. But these tools should not be confused with the analytics that can be used to extract meaning from the data.

Notwithstanding the importance of big data, its analytics—which have been evolving for a long time, even before computers came into existence—now have a huge array of methods and techniques such as data mining; machine learning; and text, audio, and video analytics (see Figure 1). These tools have become extremely powerful, and using abundant sources of data, they can deliver unprecedented business value.

Deriving Innovation from a Data-Driven Mind-set – Figure 1

Figure 1. A basic taxonomy of analytics techniques

Contributing greatly to the recognition of the power of big data analytics is the realization that with massive sets of data, sometimes brutally simple algorithms can be—paradoxically—much more effective than sophisticated algorithms in solving certain classes of problems. Two examples are machine translation and speech recognition. Google Research analyzed this realization, and it is demonstrated in applications such as Google Translate and Siri.2

A mind-set driven by data

Big data analytics can inspire fresh ways for identifying, framing, and solving data-centric problems. This approach allows the data to speak and is known as the data-driven mind-set. This mind-set starts by identifying and collecting data needed to understand a given business area and ends with evidence-based confirmation of an improvement or a solution. The data-driven mind-set can be outlined by the following activities:

  • Identifying and collecting data to explore a specific business area
  • Diagnosing the current situation and identifying areas with problems or potential
  • Framing and reframing each problem using insights gleaned from the data
  • Distinguishing possible alternatives based on relationships between data objects
  • Forecasting the impact of candidate improvements or solutions on key business metrics
  • Tracking business performance and confirmation of an implemented solution

This approach includes the following several characteristics that make it very effective in reframing problems and for providing novel, evidence-based solutions:

  • Using correlation before causation
  • Overcoming most limitations from sampling by using all data
  • Usually tolerating incomplete or dirty data
  • Applying predictive models to forecast the future
  • Having a preference for near-real-time data

The data-driven mind-set uses correlation before causation because it is good enough for many practical purposes—for example, in product recommendations. Correlation fills a very important gap between implicit gut-feel models and very elaborate causation models that may take considerable time and effort to build (see Figure 2). The correlation does not speak to why something is happening but provides an alert that it is happening.3 This characteristic can dislodge us from the grooves of our current mind-set, help us overcome our biases, and invite us to explore new ways of doing things.

Deriving Innovation from a Data-Driven Mind-set – Figure 2

Figure 2. The key role of correlation uncovered in data analysis between gut-feel and complex, causation models

For example, Google Flu Trends uses aggregated Google search data to estimate current influenza activity around the world in near–real time, which is significantly faster than estimating by health authorities who generally rely on aggregated reporting by hospitals. Another example is’s item-to-item collaborative filtering that uses correlation for making book recommendations.

Although the data-driven mind-set and the above characteristics may have existed all along, it never had the enabling technology until now. With big data analytics tools organizations can empower staff with such a mind-set to mine data from internal and external sources for insights and innovation.

Part 2 of this article turns to innovation, discusses why it can be so challenging for many organizations, and shows how big data analytics and a data-driven mind-set can spark, guide, and sustain innovation. In the meantime, please share any thoughts or questions in the comments.

1,3 Big Data: A Revolution That Will Transform How We Live, Work, and Think, by Victor Mayer-Schönberger and Kenneth Cukier, Eamon Dolan/Houghton Mifflin Harcourt, March 2013.
2 “The Unreasonable Effectiveness of Data,” by Alon Halevy, Peter Norvig, and Fernando Pereira at Google, IEEE Computer Society, March 2009.