To serve a growing customer base and better manage the client experience across all customer touch points, organisations are moving away from siloed transaction-oriented systems – such as enterprise resource planning (ERP), customer relationship management (CRM) and dealer management systems – in favour of more integrated and socially aware systems.
This shift has two effects. Organisations need to manage the surge in the type and overall volume of data, and at the same time be able to analyse a large amount of complex data in real time.
While the volume of structured data from traditional transactional business applications such as ERP and CRM continues to grow, CIOs are facing an onslaught of unstructured data from multiple sources such as social platforms, and semi-structured data from machine-to-machine (M2M) communication.
This signifies the need for traditional business intelligence (BI) approaches to supplement big data approaches, even for basic structured transactions.
Social Media Drivers
Companies are taking advantage of social media’s growing user base, using platforms such as Facebook, LinkedIn and Twitter to engage with customers directly. Other examples of online communities include consumers reviewing products at mobile app stores and third-party merchant websites.
Firms such as AT&T, Carphone Warehouse, Domino’s, Procter & Gamble, Tesco and Unilever now regularly use a variety of these platforms to engage with their customers. Data from social media helps organisations undertake sentiment analysis on their consumers and better tailor their offerings.
Businesses are increasingly using mobile devices not only to push information to consumers, but also to engage with them via services such as mobile banking and mobile wallets.
Other devices continuing to grow in usage include self-service kiosks and smartcards that automate basic processes such as checking in at airports and paying for parking. For example, one telecoms provider in Southeast Asia regularly collects location data from its customers’ mobile handsets to better understand their behaviour and thereby improve its marketing campaigns.
Gathering Sensor Data
Sources such as radio frequency identification (RFID) and sensor networks, which are typically used in B2B environments, transmit semi-structured data that firms must harness to make better business decisions.
Retailers are now able to use this data to gain insight into the demand for their products and better manage their inventory and supply chain. They can also use the data to influence decisions around new product development to improve the customer experience.
A Holistic Approach
As organisations explore technologies to complement their existing investments and support big data, it is critical to base decisions on the existing information management architecture and identify components that they can reuse and consolidate. Companies must view big data implementation as a business project, not an IT project.
Traditionally, IT-enabled business processes have largely been defined around the structured data streams of process-based apps such as CRM and ERP. But it is critical for organisations to change this to include external sources and to redefine the data acquisition process (sources and types).
Firms must start big data initiatives at the business process layer and work with business executives to better understand business events that will eventually define job and workflow management. Involving business leaders at this stage can also help IT leaders outline the key business-specific metrics and key performance indicators (KPIs) that they will ultimately use big data implementations to monitor.
Better customer data management helps reduce duplication in the database and improve the accuracy of individuals’ profiles. While the process around managing customer data has traditionally been optimised for discrete attributes such as age, gender and name, big data is changing this drastically.
Companies now need to allow customer profiles to include behaviourally oriented data, such as social networking information and mobile telephony details, and ensure that they can combine this with the traditional attributes.
Firms must expect their integration costs to grow in some scenarios, as the cost of adding new data can make traditional data integration methods too expensive.
To run analytic models for big data, organisations must invest in technologies that support massively parallel processing (MPP). It is important for senior decision makers to pick and choose technologies that don’t require them to rip and replace the existing architecture, but that will complement current investments.
The ultimate goal should be a single platform for all data warehousing requirements, as opposed to disparate sets of processing units. But in the meantime, most firms will have to govern three key pieces of their data warehouse architecture: traditional enterprise data warehouses; data warehousing appliances; and big data processing capabilities. Analysing big data requires organisations to adopt new analytical models and approaches that allow large-scale indexing of data entities and support relationship analytics to better understand the relationships between these entities.
Big data sources such as location-based data, social networking information and telecoms user details can help organisations establish relationships among data entities. Real-time analysis of these big data sources can provide companies with insights valuable for targeting and messaging. However, the new real-time analytics for big data have neither the luxury of historical data nor the time to allow for trending analysis, and only offer directional insights, so firms should expect slight degrees of variance. Instead, more traditional sources of data that do not require real-time processing are more likely to offer accurate insights for the business.
It is important for organisations to define the right use case for both types of data – the value lies in harnessing insights from both types of data to inform business decisions.
Big Data Headaches
As companies adopt new applications and approaches to cater to non-traditional touch points; they are faced with an explosion of information. This increase in data volumes poses a new set of challenges around information management and architecture.
The technical challenges of managing, accessing, and analysing real-time data streams means that IT teams are often unable to respond quickly enough to new market dynamics and improve customer experiences. Conversations with senior decision makers and a recent global survey of clients with knowledge of or experience with big data confirm that the explosion of information, both from traditional data and big data, is not just about volume.
While respondents indicated that volume is the main reason for considering big data solutions, they also indicated the velocity of change, the variety of data formats and structural variability as major concerns.