Smart Consolidation for Smarter Computing
On Monday I wrote of our motivations to develop a new strategy: “Smart Consolidation for Smarter Computing”. Following my keynote session at Enzee Universe ’11 I’m ready to share details. The Smart Consolidation strategy simplifies infrastructure, freeing enterprises to deploy analytics rapidly across new forms of data, with more varied applications to large communities of users.
We are adopting Gartner’s terminology of the ‘logical data warehouse’ to describe how organizations can realize this strategy.
IBM Smart Consolidation and the “Logical Data Warehouse”
Smart Consolidation is evolutionary, offering those invested in large EDW platforms a path to the logical data warehouse. We believe this realistic vision can be achieved in two years or less. Smart Consolidation has three key tenets:
Consolidate infrastructure to simplify analytics. Appliances and specialized systems reduce complexity by consolidating sprawling marts to a small number of systems.
Distribute data and compute to fit-for-purpose systems Computation is mapped to appliances & systems specifically designed for well-understood workloads. These specialized systems offer optimal performance at affordable prices, their simplicity accelerates time-to-value, and their deployment frees the EDW to assume a more focused role of enterprise data hub.
- Coordinate management & governance across systems IBM’s industry-leading portfolio of tools, management, and governance make the logical data warehouse simple to manage and at low cost. These policies and tools manage meta-data, provide data governance, distribute/replicate data, and manage data’s lifecycle.
Getting there is (more than) half the fun… The steps towards this vision are evolutionary not disruptive, particularly for clients who have already built EDW-centric architectures.
Step 1. Offload analytics workloads from the EDW to appliances and systems to consolidate “spread-mart” sprawl. The EDW is now the central locus for data governance and metadata management. Use simple, effective tools for data flow planning, data movement and governance. This step delivers high performance analytics and simplifies infrastructure and its management.
Step 2. Introduce queryable archiving to provide cost-effective analytics on massive sets of data at an economically-viable price point. This satisfies regulatory or law enforcement needs of many industries. At this step, the introduction of data lifecycle management retires data from the EDW and/or analytic appliances to the queryable archive system guided by rules based on age and/or frequency of use.
Step 3. Accommodate new, (e.g., “Big Data”) sources into the analytic infrastructure, using systems including real-time streaming analytic engines and Hadoop platforms to undertake pre-processing and initial analysis and ingest data to the logical warehouse. Data can then flow from those platforms to other analytics appliances and systems for further downstream processing. Certainly many of these new data sources may initially pass through very different data integration and governance filters. Once they flow beyond their platform of ingestion, the enterprise data warehouse applies governance, data flow and lifecycle rules. As new paradigms and analytic platforms emerge, they too can be similarly integrated.
Step 4. The EDW evolves into a “Data Hub” for the enterprise – incorporating data integration, cleansing, governance, metadata management and distribution of data flows to the appropriate analytical platforms. Designed for processing mixed workloads at very high throughput, IBM Smart Analytic System is an ideal platform for the enterprise data hub. A recent study by International Technology Group finds “Three-year costs for Smart Analytics System 7700 are 43 and 40 percent less than those for Oracle and Teradata systems respectively (1.)
The following chart shows some of the IBM product portfolio products as examples of how they might sit in this logical data warehouse, but the objective is to allow for heterogeneous environments where one or more of the elements shown could be provided by a vendor other than IBM.
Example Analytic Platforms for Smart Consolidation
Over the next few months, we’ll be sharing a roadmap evolution of products and features underlying this strategy that will provide our clients the flexibility to maximize the value of their investments in data warehousing and analytics while scaling to support new data types, higher data volumes and more complex applications, with appliance-simplicity. Stay tuned.
For more information on Smart Consolidation for Smarter Computing:
- Download the Smart Consolidation eBook
- New Video features an interview with Phil Francisco on the "Smart Consolidation for Smarter Computing" EDW strategy.
1. Cost/Benefit Case for Enterprise Warehouse Solutions -- In-depth Comparison of IBM Smart Analytics System 7700, Teradata Active Enterprise Data Warehouse and Oracle Exadata Database Machine; June 2011; International Technology Group.