Why analysts should take a predictive approach to problem solving

Research Director, Advanced Analytics, TDWI

Advanced analytics is a technology whose time has come. The good news is that organizations realize the importance of analyzing their data, and they are using increasingly advanced analytics techniques to do so. The bad news is that they are challenged to find the skills to perform advanced analytics.

In the recent TDWI research report Next-Generation Analytics and Platforms for Business Success, respondents were asked about their top challenges with next-generation analytics. Next-generation analytics move beyond reporting and dashboards to techniques such as predictive analytics. By far, the most significant challenge cited by respondents was the lack of skilled personnel.

Skills are needed all around for advanced analytics. This skill set sometimes requires knowledge of recent data-related technologies such as Apache Hadoop. It can require knowledge of analytics techniques and how to use them. On the analytics front, vendor tools are becoming easier to use. Interfaces are easier to navigate than previous versions. Some tools can take data and decide which models make sense once outcome variables are specified. Some tools can even determine appropriate models and then automatically put together the story as output. However, many organizations realize that this behavior doesn’t negate the need for people who can frame a problem, interpret the output of an analysis and communicate the results.

Advanced analytics skills

When organizations consider the skills they need for advanced analytics, they often come up with knowledge of the business, knowledge of the data and critical thinking. These skills are certainly important; however, there are a few others that are also important including techniques, mindset, and communication.

Understanding the techniques used, at least at a basic level, is important. Three good techniques for predictive analytics include clustering, decision trees and regression. Analysts need to understand them in the same ways businesspeople understand them. If analysts can’t explain the results of an analysis, they can’t defend it, and then people won’t trust it.

Interpreting and trying to communicate the results are equally important. If a tool is using a logistic regression, what does that technique mean in terms of results? How should the output be interpreted? Does the interpretation make sense in the context of the technique used? Vendors typically offer training for their toolset that may include some overviews of different techniques, and getting the training from other sources as well is possible. The level of training depends on how deep analysts want or need to go and what kinds of analysis they are doing.

A certain mindset goes hand in hand with advanced analytics. A specific thought process is required that is different than the reactive SQL query. For example, consider predictive analytics for churn. The business intelligence (BI) and SQL thought process could consist of reporting on customers who dropped a service. A dashboard may be available that shows how many people switched providers, how much they spent on service each month, and so on. However, the forward-looking approach attempts to predict the probability that a customer will drop a service.

The proactive, predictive approach requires a different way to think about the problem. It requires looking at the target or outcome of interest—in this case, was the service dropped or retained? Then, using relevant—and potentially derived—attributes about customers such as billing information, purchase history, length of time as a customer, demographics, even geospatial or other data such as text data, provide a training set of data to a tool. The output consists of classification rules with associated probabilities—in the case of decision trees—about the characteristics of a customer who would drop or retain a service. A test data set can then be applied to the algorithm to see how it performs against the new data. If it performs well, a model is available for use. The trick is to change the thought process and look for the characteristics of customers who behave a certain way, and not just make a query to illustrate how customers behaved.

The ability to communicate results is the key to telling a data story—a narrative that includes analysis. The story needs to move beyond the simple recounting of facts to weave together pieces of analysis that make an impact. This development can be a one-time story—that is, a presentation—or a modern story including storyboards that may be updated as the data changes. These stories need to grab the attention of the audience and get them emotionally involved. Honing communication skills is an important consideration for analysts.

Skill-building avenues

What are today’s organizations doing to build talent for analytics? Many businesses are using a combination of approaches that include creative hiring and training, along with utilizing knowledgeable consultants and partners (see figure)

Interestingly, based on recent TDWI research at least, only a small proportion of organizations hire recent college graduates, and they do so once their own team has reached a critical mass. In this way, the recent graduate hires, who might know the techniques but not the business, can learn from those who do know the business.

Current employees understand the business. Therefore, many organizations look to train from within. Some set up on-site training. Others send employees out for training classes and boot camps offered by independent firms as well as software vendors. Of course, organizations need the funding to offer this training. Many companies are trying to train themselves by using complimentary online courses.

Some organizations can build competency centers or centers of excellence. Centers of excellence typically consist of a cross-functional team that provides leadership in analytics. The team may be responsible for analysis, training, and disseminating best practices. Sometimes these centers are centralized, and sometimes they are distributed. Centers of excellence can be very helpful.

In some organizations, the data scientist—if it is one person—is often a scarce resource. Some organizations look to anchor an analytics team with a few data scientists. These people can act as mentors—perhaps participating in the aforementioned competency centers. They can also be a control point for business analysts who are building models. For example, if a business analyst builds a model to be put into production, the data scientist reviews it first.

Knowledgeable consultants and partners

Organizations may also hire consultants and other partners for advancing their analytics efforts. This method can be a good way to jump-start programs. Some consultants also provide training. For example, consultants may build and operationalize a model, but also work with the business to help make it self-sufficient.

While vendors have made huge progress in making advanced analytics easy to use, the reality is that training may still be needed. Training may include tools and techniques along with framing a problem and communication building. Some training can happen organically; some needs to occur in a more formal way. Training cannot be overlooked when planning for advanced analytics. A lack of preparation can spell doom for achieving success with advanced analytics initiatives.

Data scientists require power tools, especially predictive analytics and data mining solutions. In its recent evaluation of the predictive analytics solution market, Forrester Research said, "IBM assembles an impressive set of capabilities, putting predictive at the center. No matter how an organization wants to get started with predictive analytics, IBM has an option for them. The solution offers one of the most comprehensive set of capabilities to build models, conduct analysis, and deploy predictive applications: both on-premises and in the cloud. With customers deriving insights from data sets with scores of thousands of features, IBM's predictive analytics has the power to take on truly big data and emerge with critical insights. Read the complete report to learn:

  • Why predictive analytics is a game changer
  • The six steps of predictive analytics
  • Which vendors are Leaders, which are Strong Performers and which are Contenders