Big data challenges

Post Comment
Director, Marketing, Big Data, IBM

The business environment is shifting and a new era of computing is upon us. As with all emerging markets this can present significant challenges. Every time there is major technology or market shift, there is a real opportunity for new leaders to emerge as others stumble and fail. Transformational change is difficult and presents significant obstacles. Two of the most common challenges I hear about when talking to clients about big data are how and where.

People and process. Implementing technology to solve big data challenges is not trivial, but like most transformational projects it’s typically easier than the cultural change needed to get people to act on new insights that companies derive from big data and analytics.

At a recent IBM Big Data and Analytics customer board, one client indicated that you need to have analytics at the front-end of processes and there is a methodology on how to integrate analytics into business processes. IBM advises organizations there are a couple of things you must get right  for effective big data and analytics transformation.  

First, build a culture that infuses analytics everywhere.

How is IBM helping clients with the culture challenge? An excellent client example is that of a property, casualty insurer partnered with IBM’s Global Business Services organization. Through this partnership with IBM GBS, this client established an Analytics Resource Center and built a powerful big data & analytics platform. Their success was largely driven by their focus on taking the right approach:  Establishing the right organizational structure and culture, driving strong change management and creating training programs to ensure all users understand how to take advantage of the analytics provided to them.

Now, over 40 percent of the employee base uses analytics on a regular basis to fuel their decisions. Of those users, 80 percent use OLAP to enhance decision making to improve company performance and 95 percent of the users agree that adopting an analytical approach has improved their ability to evaluate business performance.

The second challenge is all about the question: "Where do I start?"

The danger here is that big data is so pervasive now, that there is pressure on every organization to have a big data strategy. In fact, in my last blog I wrote I noted that “if you ignore big data you risk being left behind and becoming a dinosaur.” You often hear stories of CEOs and other C-level executives telling their employees "we have to do big data" or "we have to implement Hadoop." However, most CIOs struggle to define the roadmap for big data and demonstrate early value. 

As with all IT projects, there must be a business problem you are trying to solve; identify that problem or set of problems. Do you want to improve the customer experience by better understanding customer preferences and behavior? Or, do you want to use big data technologies to significantly improve intelligence, security and law enforcement insight to predict and prevent crime? The abundance of growth in machine data is another major driver of big data. Is there value in your organization using sensor data for predictive asset maintenance?  

While I believe it’s fundamental to have a business problem you are solving to get big data projects funded, another approach to starting is for your developers to experiment with new and emerging technologies like Hadoop or stream computing. While testing out new technologies, new potential uses that you didn't previously have in mind are sometimes uncovered.

To help clients get started, IBM offers various approaches to meet you at the point of your need. These include IBM Big Data Stampede and IBM Quick Start. “Big Data Stampede provides a vehicle to enable clients to accelerate their big data journey. We fuse the strength of our services, products and skills to minimize the barriers of big data initiatives and provide clients with quick success. Big Data Stampede is a solution that provides customers with expert resources at cost to help them get started with big data and promote usage of big data technologies. We work with clients to educate them on how to get started quickly and receive value from big data.

IBM also offers Quick Start versions free of charge for IBM InfoSphere BigInsights for Hadoop and IBM InfoSphere Streams products. These are downloadable editions that allow you to experiment with easy-to-follow tutorials and videos. There are no time or data limitations which allow you to discover how to apply these technologies to a wide range of use cases.

What do you think? What challenges do you see in starting big data initiatives? How did your organization start its first big data project? What business problem(s) are you solving with big data?