Developing on-demand streaming analytics in the cloud

Big Data Evangelist, IBM

Streaming analytics is central to businesses everywhere. One can even argue that streaming architectures are as fundamental to today’s live, cloud-oriented data services as relational data architectures were to the prior era of on-premises database computing.

This trend explains why, for example, growing interest exists in approaches, such as the so-called Lambda Architecture, for doing justice to both in-motion data and stored data at-rest in the cloud. For those unfamiliar with this concept, it refers to the need to integrate both batch and stream processing within a common architecture under a common development, runtime and administration paradigm.

Growth of on-demand business analytics and cloud analytics are increasingly intertwined in this era of on-demand business. Enterprise requirements for on-demand, cloud-integrated streaming analytics continue to grow. According to one independent market research firm, the market for streaming analytics is expected to grow to almost $2 billion by 2020, representing a 31.3 percent compound annual growth rate (CAGR).

Who may benefit from an on-demand, streaming analytics service in the cloud? For starters, anyone who’s implementing Internet of Things data stands to benefit, which depends closely on streaming and distributed cloud architectures and pervasive analytics. Check out my recent IBM Big Data & Analytics Hub blog posting on the changing contour of big data analytics in the era of the Internet of Things.

Many industries—including mining, manufacturing and healthcare—can benefit from Internet of Things data and other streaming, machine-data analytics environments for real-time demand forecasting, preventive maintenance and closed-loop operational efficiencies and remediation. And then consider all the established, industry-aligned streaming analytics use cases. Many of them are either entirely cloud-centric now, or they are rapidly heading in that direction.

For example, the advertising industry—and any e-commerce or other site that derives revenues from online ads—can benefit from cloud-based streaming analytics to aggregate ad data, monetize location-tagged customer data, extract social media insights and drive smarter digital campaigns. Likewise, the gaming and media industries can benefit from streaming analytics of user-generated data, web and social log data, and sentiment intelligence. In the automotive industry, connected cars are expected to rely on streaming cloud analytics for congestion management, real-time traffic updates and other critical intelligence and coordination.

Tools of the trade

Developing streaming analytics for these and other cloud-based use cases requires tools and platforms that have been optimized for such challenges. At the very least, development environments need to provide several capabilities:

  • Intuitive toolkits that define the intricate, predictive logic that drives real-time, high-volume, low-latency ingest, analysis and correlation of millions of pieces of diverse data
  • Visualizations and analytics that offer interactive viewing and analysis of data that flows from and among thousands of streaming data sources in public, private and hybrid cloud environments
  • The ability to promote streaming analytics applications and instances into in-production cloud environments
  • Assurances that the target cloud environments provide the requisite security, availability, disaster recovery, multitenancy, monitoring, metering, resource management, scalability and elasticity

Check out IBM Streaming Analytics for Bluemix, which is now in beta and delivers these benefits. You can also benchmark performance numbers on the service. And a 451 Research report on a sibling offering, IBM Streams 4.0, is also available by registering for the download.