IBM’s “Smart Consolidation” – Rethinking the EDW

VP Product Management & Marketing

This week at the IBM Netezza Enzee Universe show in Boston, we’re taking the covers off IBM’s strategy for data warehousing and analytics and we’ll be talking about what that’s going to mean for the joint product portfolio.

“Smart Consolidation” aims to provide IBM clients, customers and partners with a simple, scalable and appliance-oriented way to deal with the challenges of today’s analytic workloads. In the past six months, I’ve sat through sessions at IDC, Forrester and Gartner events and multiple webinars, blog postings and twitter streams discussing these challenges.

While it may have had some success in the past*, the concept of the single, monolithic, all-knowing, enterprise data warehouse to run the entire business is now an anachronism: the variety of data types and applications, and imperatives for businesses to become agile expose the inflexibility and brittleness of the old “one system fits all” approach.

Enterprises recognize analytics and new data sources as vital tools in a highly competitive market. Social networks and machine-to-machine communications create a lot more data, but the challenges go beyond data volumes:

  • shrinking tolerances on data latency and on the time required to reach key business decisions;
  • increased scope of analytics;
  • increased complexity of analytics;.
  • increased variety of data.

An inflexible EDW responds too slowly to changing business requirements: new applications projects and data sets take weeks, months or even years to implement; and significant IT resources are devoted to daily care and feeding of the warehouse – like “Scotty” in Star Trek, increasingly called upon to “squeeze more performance” out of a system that is already being asked to perform beyond its red-line.

For years enterprises have attempted to work-around the problems, offloading data and computation to “spread marts”, secondary platforms encompassing spreadsheets to multi-terabyte data marts and everything in between, each often only “loosely affiliated” with the central data management of the EDW The resultant “sprawl” translates into cost, duplication technology and governance issues that further drain the limited resources of the IT professional staff.

With the acquisition of Netezza completed in the fourth quarter of 2010, IBMers and the Netezza team sat down to discuss how, with the potent product portfolio at IBM’s disposal, we could address these issues. The strategy needed to deal with the current challenges of analytics and warehousing, be flexible and accommodating to those of the future. And it needed to be simple – reducing the operational burden and providing a faster return on our clients’ investments. The strategy we came up with is called “Smart Consolidation”.

What is Smart Consolidation? Well that’s the topic of my next posting – check back here in a day or so.

[*: at the Gartner Business Intelligence Summit held in Los Angeles in May of this year, Gartner vice president and distinguished analyst Donald Feinberg not only declared the EDW concept “dead” but also suggested that it had “never” really worked.]