Why big data?

Making the complex simple

Managing Director of Intelligent Business Strategies Limited, Intelligent Business Strategies Limited

Everywhere you go today, people are looking down at a mobile device. They are online, browsing, collaborating, shopping for goods and services and transacting business. And not just consumers are using them. Mobile devices are also being used extensively in business-to-business transactions.

Customers and prospects have become empowered in the online world. Social networks and comparison and review websites have allowed them to quickly become well informed before they buy. With information on their side and at their fingertips wherever they are, mobile device users have the power to sacrifice loyalty in the blink of an eye or the click of a mouse if quality and services are not satisfactory. 

In such a fast-moving, online, mobile economy, in which company lifespans are shrinking, the rise in customer-focused CEOs is not surprising. This focus is about business survival; companies need to retain and grow customers, optimize business operations, reduce risk and improve financial management. Business survival is becoming as much about customer retention as it is about growth, and if companies are going to survive they need to know as much as possible about their customers.

New data sources

What do you know about your customers? Much of what companies know is typically held in a data warehouse—a database that collects transactions and looks at customer transaction activity over time to understand who is buying what through which channel. Unless a transaction takes place, you know very little. And with the competition breathing down your neck, is it possible for you to get data beyond transaction data, analyze it and enrich what you already know about customers? The answer of course is yes. We can get a lot of data from several new sources: 

  • Clickstream data: Analyzing all the clicks of every visitor on a website fosters understanding site-navigation behavior, the paths people take to buying products and services, what else they looked at on the way to buying, paths that led to abandonment and more. This information helps improve the customer experience and conversion. It may also be possible to associate clicks with customers and prospects. 
  • Shopping cart data: This data from your website enables you to see what people are putting into and taking out of shopping carts en route to online checkout. 
  • Social networking data: Analyzing data from Facebook, LinkedIn and Twitter gives you an opportunity to obtain information about customers you don’t yet have. It also allows you to identify previously unknown relationships, what people like and dislike and so on. Analysis of this data also enables you to identify who the influencers are in the network and which people link multiple communities in the network. Targeting influencers with marketing campaigns could significantly boost sales. Analyzing social network data also empowers you to identify sentiment information—what people are saying about your products, your customer service and your brand. 
  • Sensor data: Analyzing data from smart technologies such as Global Positioning System (GPS) sensors in smartphones can give you information on product usage or location. Sensor data may also be available for monitoring production lines, asset performance, supply chains and distribution channels to see if customers are getting deliveries on time. 

Much of these data types typically have characteristics that make fitting them in the traditional tabular rows and columns of relational databases (RDBMSs) challenging. Text data, for example, is unstructured data; but we still want to analyze it to extract details about people, products, locations, monetary amounts, dates and times, and to understand sentiment.

Today we are analyzing structured data in the form of transactions, semistructured data in the form of JavaScript Object Notation (JSON) or XML and unstructured data in the form of text and images. The data volumes can be very large, and capturing and analyzing high-velocity streaming data at scale is rapidly becoming a requirement for organizations.

Advanced types of analysis

Because of big data’s complexity, advanced, highly complex types of analysis beyond traditional analytics workloads in data warehouse systems such as ad hoc query and reporting, online analytical processing (OLAP) and visual data discovery are now needed. Data is more complex, larger in volume and arriving more quickly than ever. It requires advanced analytical workloads: 

  • Analysis of data in motion—streaming analytics
  • Complex analysis of structured data
  • Exploratory analysis of data not modeled and multistructured such as Twitter data for sentiment analytics
  • Graph analysis such as for social networks 

The result has been the emergence of new platforms that are well suited for these new analytics workloads that extend the analytical environment beyond the data warehouse into multiple types of data stores.

These platforms are shown in the diagram above and discussed in upcoming blogs in the “Making the complex simple” series.