A Different Methodology for Big Data

Get acquainted with minimum viable insight for moving analytics-driven big data projects forward

Solution CTO, IBM

One recurring topic in the big data space is, how do I get started? This column has covered that topic before, but I want to expand on my last post on the subject to be more prescriptive about how to get started when your organization is still on the fence about these technologies. I'd like to introduce a new way to get started that leverages the methodologies previously laid out but extends the approach into a construct that helps move big data projects forward. Moreover, it moves them forward consistently with the pragmatic- and return on investment (ROI)–focused guidance I’ve provided previously. Some may be familiar with the idea of minimum viable product (MVP) that has been all the rage recently in Silicon Valley. MVP is an approach popularized by Eric Ries, author and Silicon Valley entrepreneur, for web-based start-ups dealing with fast-moving markets. The basic idea of MVP is to ground activities based on market acceptance and be highly customer driven on the products that are introduced to the market. The following key reasons are why Eric advocates MVP:

  • The market moves too quickly for traditional, very long lead times.
  • Feedback from customers trumps paper modeling.
  • Trying to fully bake the production use cases and usage patterns before delivering them is a fool’s errand.

If you take a step back, the challenges that MVP attempts to address sound awfully familiar to those of us who work with innovation in complex enterprises. This familiarity made me think: Why not introduce a similar approach for successful big data projects in which an agile-based method of delivering proof of value helps get people on board with new ways of doing things? Therefore, I’d like to introduce the idea of minimum viable insight (MVI). MVI basically considers the minimum hurdle that validates a new approach to problem solving by delivering insight that hasn’t been possible before. MVI works by taking a flexible and agile-based approach to validating a methodology for solving problems that provides an “aha” moment of insight. The insight has to be intuitively valuable for MVI to work, so it helps organize a focus on real problems to be solved. For example, and sticking with my “don’t boil an ocean, pick a bathtub instead” analogy, we recently wrapped up the first phase of a transformational journey with a financial services organization. We consumed 36 months of highly diverse data and transaction sets and used our Apache Hadoop–based IBM® InfoSphere® BigInsights™ cluster to work with the data without requiring any preplanning on which analytics we were going to run. The idea was to prove we could tap into the patterns hidden in the data without having long lead times, thereby showing both the business and IT teams that there were alternative ways to quickly sprint to business value. Evidence of our success would be insight that this firm had never generated, which—to be sure—passed my ROI hurdle. And I think more importantly, it was intuitively understood by senior management to be valuable—in other words, we needed to deliver MVI. MVI needs to be intuitively valuable; there are no points for delivering an insight that is technically interesting but cannot be seen to move the needle. It also needs to be something that brings the larger opportunity into view when an executive stares at it. There were several substantial opportunities, actually, but in the end we concentrated on one finding to anchor the executive readout. I can’t get into the actual pattern, but in summary we found a major signal that this firm’s customer relationships were going to change and showed the 12-month impact of that change occurring. There were literally more than 10,000 different ways to slice the data, but we stayed focused on an MVI-based strategy and used that approach to galvanize support for the next phase of the project. Of course, MVI doesn’t exist in a vacuum, and it needs to support and fit into the expanded strategic imperatives that most organizations are facing. By design, the MVI approach is specifically focused on helping provide a path forward from today to tomorrow for large issues that are analytics-driven. Today we largely try to plan everything out, try to perfectly model everything into the future, delay actually working with the data for months to years, and design things to remain the same. For tomorrow we should be learning through doing (not planning), accepting that uncertainty is the rule, designing for change, and prioritizing learning from direct experience as quickly as possible. The next installment will address the MVI approach in more detail, but hopefully this introduction to the concept of MVI offers some rationale for why it is useful. Please provide any questions or thoughts in the comments.   [followbutton username='thomasdeutsch' count='false' lang='en' theme='light']

 <table cellpadding="0" cellspacing="0" valign="top" width="15%>

  [followbutton username='IBMdatamag' count='false' lang='en' theme='light']