Exploring Streaming Data in Real Time

Turn streaming data into instant actionable insight by applying visual data discovery to data in motion

Manager of Portfolio Strategy, IBM

The world of business intelligence (BI) and analytics is undergoing significant changes, and it’s about time. Streaming data holds the potential to dramatically reduce the time to deliver business-critical information if it can be processed, understood, and analyzed in real time. Visual data discovery enables line-of-business users to independently and easily create and share analysis without delay.

Both streaming data and visual data discovery technologies are independently having an impact, but new advances are now allowing both to be used in conjunction and represent a breakthrough for big data analytics. Streaming data alone doesn’t help organizations, but taking action on streaming data using rigorous analytics and visualization delivered in milliseconds can turn streaming data into actionable insight. The key is bringing the analytics and visualization to the data rather than first asking questions and then finding the data that answers those questions.

Displacing traditional business intelligence

The move to visual data discovery started gaining traction in 2009 as business users began to grow increasingly frustrated with the long delays and inflexible, static reporting delivered by IT. Today, the shift to a visual data discovery approach is gathering tremendous momentum.

According to the Gartner Predicts 2014 report, “Companies will shift their future investment away from IT-developed reporting solutions toward business-user-led analysis solutions. IT will focus more of its effort on system-of-record data modeling and governance. As a result, data discovery will displace IT-authored static reporting as the dominant BI and analytics user interaction paradigm for new implementations by 2015.”1

Traditional BI has been a relatively slow process as it required a top-down, IT-led approach in which the data needed to be modeled to create semantic layers, and reports were created and pushed to users. Visual data discovery delivers much faster results than traditional BI as it takes a bottoms-up approach in which business users are in the driver’s seat and can easily access and combine data from multiple sources. Instead of viewing static, tabular data from traditional BI reports, the interface is based on interactive visualization for intuitive exploration.

Traditional BI is good at answering common questions such as, “What were our sales for the quarter in Asia?” However, visual data discovery is about finding things that are not yet known and answering questions never considered such as, “I didn’t know that we closed some very large deals in Singapore, but the rest of Asia doesn’t seem to be doing well. Let me see what’s causing this anomaly.” Even advanced data discovery solutions do not support the requirements brought about by big data—specifically the use of streaming data sources.

Analyzing streaming data—the new normal

The amount of streaming data being generated is exploding rapidly from the growing variety of interconnected machines, devices, sensors, and consumer content. And the amount of data generated from any one source can be staggering. For example, Virgin Atlantic has a new fleet of highly connected planes that generate a half terabyte of data on every flight.2

Streaming analytics has long been a requirement for capital markets as stock transactions are measured in microseconds and portfolio risk changes rapidly. As advanced streaming infrastructure such as the IBM® InfoSphere® Streams analytic platform is being rapidly deployed, other industries are beginning to reap its benefits, including telecommunications, energy and utilities, healthcare, law enforcement, and manufacturing.

Gartner also predicts that by 2017, over 50 percent of analytics implementations will use event data streams generated from instrumented machines, applications, and/or individuals.3

One big benefit of combining streaming data with visual data discovery is the ability to strike while the iron is hot. Seeing live, actionable information in an intuitive visual display helps organizations monitor precisely how their business is doing at that instant. However, once someone in an organization such as the chief marketing officer (CMO) is given real-time visibility, very quickly questions emerge that require fusing historic data with real-time data. For example: “How are the online sales today different than yesterday’s sales? Last week’s sales? Last month’s sales? And which digital marketing campaigns are driving the most revenue?”

This kind of scenario led to a new generation of visual data discovery solutions such as Datawatch that can consume data streams directly from sources such as InfoSphere Streams. Streams constantly and instantly pushes information into the system, in real time, on a tick-by-tick basis. Line-of-business users can see exactly what is happening as it occurs and know precisely how the business is performing (see figure). When an outlier or anomaly is spotted, this streaming data can be combined with the historic data to get a comprehensive understanding that enables making better and faster decisions than ever before and taking action. Alerts can also be set to notify business users when certain conditions or thresholds are reached.

Exploring Streaming Data in Real Time

A Datawatch dashboard example based on data streamed directly from sources in real time

Preparing for change

Visual data discovery and streaming analytics represent the future of big data analytics initiatives. Organizations can begin to prepare for the changes today by deploying a streaming infrastructure such as the InfoSphere Streams Quick Start edition and the Datawatch Desktop trial edition.4

Please share any thoughts or questions in the comments.

1,3Gartner Predicts 2014: Business Intelligence and Analytics Will Remain CIO’s Top Technology Priority,” Gartner, Inc., G00258063, November 2013. The report is available from Datawatch.
2Boeing 787s to Create Half a Terabyte of Data per Flight,” by Matthew Finnegan, Macworld, March 2013.
4 Complimentary versions of IBM InfoSphere Streams Quick Start edition and Datawatch Desktop trial edition are available for download.

[followbutton username='madiakc' count='false' lang='en' theme='light']
[followbutton username='danpottertweets' count='false' lang='en' theme='light']
[followbutton username='IBMdatamag' count='false' lang='en' theme='light']