Aim higher: Analytics and information management in the cloud
In a previous blog post, we highlighted several metrics and expectations for future cloud adoption. These trends apply to all software categories, including enterprise applications, application development tools, databases and analytics software.
This latter category is experiencing especially rapid growth in cloud adoption, partly because of a later start in mass market adoption than in segments such as sales automation or collaboration applications. The adoption of cloud BI, analytics, database and data integration solutions began to accelerate in 2014 and 2015 saw an influx of cloud-based Analytics and Information Management (AIM) solutions. With the supply constraint lifted, and a strong demand across the market, IDC raised expectations for adoption of cloud AIM solutions from 3x to 4.5x faster than for on-premise solutions.
IDC's market research data speaks for itself. For example, in the database market, cloud database spending is growing 12x faster than spending on on-premise options. This growth is especially pronounced in the so-called NoSQL database market, where technologies such as graph databases, key-value stores, document-oriented stores, and wide-column stores exist. According to IDC's latest CloudView Survey, about 1/3 of all organizations that currently use cloud services indicate subscribing to some database services on the public cloud. That’s not to say that all databases for this third of organizations are in the cloud, but rather that the process of migration to the cloud is well on its way.
But cloud adoption is not just about software solutions. It is also about data or content accessible via cloud services. IDC refers to these as value-added content (VAC) services. Consumer demographics, buyer behavior data, weather data, data from government agencies, social media usage data, credit scores, and myriad industry-specific data sets are now available and easier to consume. In a recent IDC study of 500 organizations, 67 percent said that they currently purchase or subscribe to external data and these are not just ad-hoc efforts to enrich internal data. Well over half of respondents said their organization has a strategy or clear guidelines for licensing 3rd party information and open data. Much of this demand is driven by the need to enrich and augment internal data to provide a more complete data set as input for analytic models.
In another IDC survey conducted in 2015, we found that 77 percent of organizations in the US expanded the number of data types and/or sources being analyzed in the past 12–24 months. That’s a big change. Clearly, organizations across industries want to have a more comprehensive view of customer, operational, supplier, or other data that, along with analytics, can provide differentiation in decision making and subsequently in organizational performance. They want this data on-demand, and they want to be able to trust it. Yet, according to IDC's 2015 Big Data and Analytics MaturityScape Benchmark survey, metrics on data quality, trustworthiness, completeness, and timeliness leave much room for improvement.
Modern cloud solutions for analytics and information management will play a significant role in enabling and simplifying self-service access to the right data at the right time by all those who need it and have a right to use it. But this approach will require a process led by a strategy that takes into account the emerging modern data architecture – one that has a place for a range of fit-for-purpose technologies such as NoSQL databases, Hadoop, MPP relational and non-clustered relational database management systems, as well as streaming data management software.
Combining all of these demand and supply trends suggests that the complexity of managing on- and off-premises data and technology will increase in the short-term. For example, adopters or current cloud AIM solutions complain about poor data quality, substandard data transfer rates both to and from cloud systems, limited API access to and from cloud systems, and lack of availability of cloud system test environments. In each of these cases about 95 percent of organizations say that there is some impact from these issues on their cloud deployments.
As the latest iterations and versions of cloud AIM solutions have come to market, these issues as well as those centered on scalability, performance, security, and manageability are being addressed. Given the potential spread of data across on-premise and cloud, internal and external sources, these new cloud AIM platforms are emphasizing functionality for data governance and data lineage management—all of which have become some of the most frequently cited workloads for data integration technology as cited in IDC's 2015 survey on this topic. In addition, these platforms help bring together a range of discrete (often open source) tools for developers, database, and systems administrators, while, in parallel, automating an ever growing percentage of routine administrative tasks. IDC predicts that cloud is the catalyst for significant shifts in IT staff talent priorities and datacenter locations. By 2018, 65 percent of companies' IT assets are expected to be located offsite in colocation, hosting, and cloud datacenters, while one-third of IT staff are expected to be employees of third-party service providers.
To drive the competency and maturity of your organization in embracing cloud analytics and information management solutions, it will be important to:
- Understand the core components of a particular cloud service. Although cloud services obscure the technology component details from the end user, it is important to understand the core components of such services. Many emerging cloud platforms are based on or have major components that are open sourced. Engage with your preferred cloud service provider to assess their commitment to open source and the value-add they provide with their commercial solutions. While doing so, assess the feasibility of migration from any given cloud solution provider to mitigate the risk of vendor lock-in.
- Focus on data integration, including data preparation, data lineage, and data governance requirements of hybrid cloud deployments. Build a business glossary and map data being managed and moved to its elements. Otherwise, the complexity will become overwhelming.
- Assess the performance of the current generation of analytics and information management technologies 'born on the cloud.' Some of the cloud technologies have been redesigned to handle cloud workloads while others have been built for them from the start. Both may have a place in your data management architecture, but their performance and total cost of ownership characteristics will need to be assessed.
- Enable self-service for all constituents—business analysts, data scientists, developers, data stewards, database administrators, and so on. Self-service does not have to mean a loss of control. On the contrary, best practices have shown an effective use of organizational structures and business process that combine shared services with distributed expertise, leading to higher data integrity and actionable data across the enterprise.
ABOUT THIS PUBLICATION
This publication was produced by IDC Custom Solutions. The opinion, analysis, and research results presented herein are drawn from more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Custom Solutions makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee.
COPYRIGHT AND RESTRICTIONS
Any IDC information or reference to IDC that is to be used in advertising, press releases, or promotional materials requires prior written approval from IDC. For permission requests contact the Custom Solutions information line at 508-988-7610 or email@example.com. Translation and/or localization of this document require an additional license from IDC.
For more information on IDC visit www.idc.com. For more information on IDC GMS visit http://www.idc.com/prodserv/custom_solutions/index.jsp.