Deriving Innovation from a Data-Driven Mind-set: Part 2

Discover how big data analytics can spark, guide, and sustain innovation

Part 1 of this article provides a detailed overview of big data analytics and the mind-set organizations can adopt for inspiring new ways to identify, frame, and solve problems in a data-centric approach. This installment looks at how this approach can guide the process of using innovation efficiently, effectively, and predictably to solve problems and track their implementation to help advance business growth.

Innovation is more than incremental efficiency improvements. Innovative ideas may disrupt established processes instead of just tweak them. Of course, every organization should strive to leverage innovation to help improve the efficiency of its processes incrementally and to introduce innovative ideas that could dramatically alter its business model.

Understanding the challenges of innovation

Innovation does not simply mean new or different. As Craig Federighi, senior vice president of software engineering at Apple, said in a recent Bloomberg Businessweek article, “New? New is easy. Right is hard.”1 Although conceiving new ideas may be challenging, coming up with a fresh idea that also proves right can be even more difficult. Ideas that are new, practical, and beneficial are called innovative ideas. Some reasons why innovation can be challenging include the following:

  • The power of the established model: Dislodging established ideas, models, and processes can be challenging because of the mind-set, “this is the way we have always done it.” Innovation often includes unlearning and undertaking a creative destruction of prevailing ways of thinking and doing things, which some people—particularly those who benefit from the established model—may find threatening.
  • An inability to handle incoherence: People often tend to filter out too early new ideas that are inconsistent with current models of explanation.
  • Following experts: People also rely too heavily on experts, but experts can perpetuate the status quo, as Nassim Nicholas Taleb argues in his book, The Black Swan.2
  • No champions: According to Peter Drucker, writer, professor, and management consultant, anything that has lasted for a period of time is often perceived by management to be normal and something that will go on forever. Further, Drucker said anything that contradicts what we have come to consider as a law of nature is then rejected as unsound. Would-be champions need considerable political skills to navigate the established power hierarchy.
  • Adoption: Innovation is one thing, but having the innovation adopted is another thing altogether and can be much more challenging.

The following three basic questions characterize these barriers to innovation:

  • How can we come up with credible new ideas?
  • How can we test new ideas for validity and impact and get them adopted?
  • How can we track changes during and after implementation?

Big data analytics helps answer these questions. Of course, big data analytics is not the only way to innovate. There are other ways to foster innovation, but big data analytics is the new powerful tool in the shed. Big data analytics can spark, guide, and sustain innovation and thus help improve the efficiency and effectiveness of the innovation process and make adoption easy:

  • Spark: Innovation can disrupt the complacency of current models by listening to the data. In other words, it identifies issues and triggers the generation of new ideas.
  • Guide: Innovation allows modeling of what-if scenarios to understand the impact of advanced ideas, which facilitates their comprehensive evaluation. It also enables reducing the risk that is inherent in innovation—such as the impact of a given action on business results—and convincing skeptics through powerful, evidence-based logic of the value of adopting innovative ideas.
  • Sustain: Innovation facilitates monitoring key performance indicators (KPIs) to track and understand the day-to-day impact of applying new ideas, which hopefully encourages additional innovation.

Sparking ideas

How do organizations come up with new ideas? Most of the time fresh ideas occur from happy accidents or by using unreliable techniques such as brainstorming. Big data analytics helps generate ideas by exploiting existing or new data sources and analytics to develop novel insights, particularly by answering queries posed by imaginative minds.

Traditionally, data is collected to test hypotheses. However, data can also be very useful for exploration and unguided discovery—even before collecting the data. Questioning what data is available, what data can be collected, or what aspect of a process can be instrumented helps reframe problems. Instrumenting a process means simply collecting more information than was previously available about the process at hand.

For example, tracking information about website interaction can reveal where customers abandon shopping carts or where visitors are clicking on parts of a web page that are not linked—but perhaps should be. Another example is using radio-frequency identification (RFID) or global positioning system (GPS) signals to track movement of stock items.

New data sources can also include external data such as social network profiles and social influencers; open data sources such as census data and; and many emerging data aggregators. One example is detecting sentiment and concerns from social media data.

Once data is identified and available, exploratory data analysis may reveal unexpected structure in data, and such discoveries often suggest other ideas worth studying. For example, exceptions, gaps, and outliers in data visualization tools may offer obvious insight triggers. Well-established data mining techniques can explore data to generate ideas—to be verified—about relationships between key metrics that are important to the business, and which parameters can be influenced.

Another way big data analytics helps in generating new ideas is by tapping outside the group, department, or organization in what is usually referred to as crowdsourcing or open innovation.

Reducing risk inherent in innovation

A key problem with innovation is how to distinguish, in an efficient way, the truly beneficial ideas from the potentially large number of ideas that can be generated. Although not killing new ideas too early is important, not spending too much effort on an idea that may not lead to a practical, realizable value is prudent. How do you know the difference? Big data analytics can help.

Big data analytics helps assess—through a kind of hypothesis testing or what-if analysis—if a new idea can have a positive impact on KPIs and whether it is significant and worth the effort to implement. Predictive analytics techniques, in particular, are key to estimating the impact of changes on key metrics. This approach is usually referred to as fail fast or fail early. By predicting likely impacts based on historical data patterns or early trials, waste can be reduced and funds can be diverted to ideas that are more likely than others to show acceptable results.

Crowdsourcing represents another way to test ideas. Getting an array of positive and negative criticism can be used as another factor in the hypothesis testing.

Enabling ongoing adoption of innovation

Big data analytics may help after the adoption of an idea by tracking its impact and validating that its impact is consistent and still beneficial. Some things are inherently easier to measure than others. For example, an objective such as improve revenue is easier to measure than intangible objectives such as enhance employee morale. Therefore, the data-driven mind-set always aims to set measurable outcomes of any new actions.

Big data analytics tools used for this purpose—such as business performance management tools—are very mature and are usually in wide use, especially in large organizations and regulated industries. If these tools are in place, they can be used to measure and track the impact of new ideas on KPIs.

Performance data can also be collected and aggregated appropriately to track changes. In either case, the performance measurement scheme should be designed to measure the appropriate KPIs continuously and should be linked directly to the instrumentation of the target process and not through indirect manual reporting.

If the innovation is related to customers, big data analytics offers very effective measures using social networks to gauge it passively through market sentiment or through actively soliciting feedback from customers. Big data analytics can also be used to measure and track the innovation process itself to measure its effectiveness and return on investment (ROI).

Overcoming obstacles to innovation

Big data analytics helps overcome barriers to innovation in the following three ways:

  • It sparks innovation by promoting a data-driven mind-set that looks for and listens to the data for new insights.
  • It guides innovation using data-driven hypothesis testing.
  • It sustains innovation by using ongoing evidence-based business performance management.

There are many well-justified reasons why organizations should develop big data analytics capabilities. The contribution to innovation is not just a bonus but also an integral part of the data-driven, big data era. Business solution professionals need to apply the data-driven mind-set to their most demanding issues to see how big data analytics can shed new light on problems, verify solutions, and track their implementation.

The concluding installment of this three-part series focuses on some real-world case studies that demonstrate ways in which big data analytics has been applied to developing innovative ideas. Please share any thoughts or questions in the comments.

1Apple Chiefs Discuss Strategy, Market Share—and the New iPhones,” by Sam Grobart, Bloomberg Businessweek, Bloomberg L.P., September 2013.
2 The Black Swan: The Impact of the Highly Improbable, by Nassim Nicholas Taleb, Penguin Books Limited, 2007.