Top 3 ways to measure the success of your analytics investment

Post Comment
Product Manager – IBM Decision Optimization, IBM

Line-of-business (LoB) stakeholders want to know that their analytics investment will help them make better, faster, and smarter decisions, with measurable business results. But for many, measuring success from applying Machine Learning and Decision Optimization is not obvious. Learn the top 3 factors to look at when evaluating these powerful technologies for your business decision-making process.

The first factor in evaluating success of an analytics investment, is the quality of the resulting business decisions, typically measured in terms of Key Performance Indicators (KPIs) such as profit, customer service, or cost. For a manufacturing planner, a “better” decision (plan or schedule) might be measured in terms of efficiency, lead times, or inventory costs. For a financial planner, a “better” decision might be measured in terms of expected payback on a set of investments. For a hotel pricing manager, a “better” decision might be measured in terms of revenue and room occupation levels. 

But how can you measure the KPI resulting from the decision, before actually executing the decision? The key to getting this right, is to capture these KPIs in Decision Optimization models, where they directly reflect the impact of alternative decisions, plans, and schedules. With Decision Optimization you can not only measure the expected effect of decisions and data on KPIs, but you can also find the best decisions which optimize your selected KPIs. Add Machine Learning to this, and you can now create “what-if” scenarios to analyze not only the effect of different decisions, but also varying business conditions such as demand forecasts.

The second factor is speed. Whom among us cannot do with some more time? I often come back to a story from my undergrad engineering days, where we spent an entire weekend using simulation to find the very best heat exchanger design. The quality of our decisions was excellent – we designed the most efficient heat exchanger and won the class prize. But the speed of decision-making was terrible – we missed our entire weekend trying out different combinations of design variables! The next year I started grad school, and discovered that I could’ve solved the very same problem at the click of a button, had I only applied a Decision Optimization model.  

This optimization “magic” extends to massive enterprises – a major aircraft manufacturer recently told the story of how their scheduling reduced from several hours to just a few minutes when they adopted Decision Optimization, with 1 million simultaneous activities scheduled! Faster scheduling means shorter lead times and quicker time to market – a significant competitive advantage.  This is also typical in supply chain planning – compare tedious manual or trial-and-error approaches to a 1-click approach when you have an optimization model at hand.

The third and often overlooked factor is robustness. A decision, plan, or schedule is robust when it continues to yield good results even when the world around it changes. For example, in the case of planners who manage large utility networks, robustness is very important. They cannot afford breaks in service when there are instabilities in demand or supply due to, for example, unstable weather. Here again you need to leverage the interplay between Machine Learning and Decision Optimization to measure and improve robustness of decisions. In this case, use Machine Learning to come up with a realistic set of scenarios, such as demand forecasts, representing the system’s potential instability. Then input the set of forecasts, or scenarios, to a Decision Optimization model capable of considering the variability to find plans which emphasize robustness while yielding good KPIs.  Robust decisions are smarter decisions:  they learn from the past and from uncertainties to recommend actions which can withstand instability.

Figure 1 summarizes these 3 factors for better, faster, and smarter decision-making – by combining machine learning and decision optimization you can optimize all 3 in a single decisioning model!  How amazing is that?

Figure 1: Measure analytics success with quality, speed and robustness of decisions

Learn more about IBM Decision Optimization and how it can add value to your business.