Distilling the Signal from the Noise with Nate Silver

Social Media Manager and Webmaster, Accessibility, IBM

We all need an attitude adjustment when it comes to analytics.

According to Nate Silver, keynote speaker at IBM’s recent Information on Demand (IOD) conference, New York Times contributor, and author of the recently released book, “The Signal and the Noise: Why So Many Predictions Fail, but Some Don't,” people need to change their overall philosophy when it comes to using data in a better way.

“I want people to know how important they are in this process, but they need to develop new skills,” said Silver. “It’s less about gathering more information, but taking the information we already have and learning how to distill the essence – the signal – from the noise.”

And, it’s about helping develop these analytical skills early in the process – for those in high school and college. In fact, Gartner released a report revealing that big data is going to create 4.4 million IT jobs globally by 2015.

Silver admits he always zoned out in his calculus courses, but loved probability and statistics because they’re more hands-on. “I wish probability courses were taught by giving students real data so they could experiment more. In fact, having kids run a fantasy baseball league might train them to make actual decisions in different ways.”

Once the necessary analytics skills and strategies are in place, Silver says it’s then important to incent employees to be accurate even though layers of management make it difficult for people to deliver bad news.

Often, it’s the people inside of an organization that make it difficult to build an effective analytics program – politics, egos, lack of collaboration or old-school thinking of not wanting to understand the art of the possible.

“Encouraging people to be open and honest about what they’re doing is important,” said Silver. “For me, honesty means you can communicate in terms of probabilities. ‘I don’t know’ is a better answer to give than a false certainty.”

That false certainty can often manifest itself in something Silver calls, “Know what you don’t know.” He used an example of the Pearl Harbor attack to illustrate overconfidence in data. Even though the U.S. had ships patrolling the Pacific Ocean, they were unable to cover the entire sea thereby creating blind spots in the analysis.

To this point, Silver also suggested that everyone needs some humility when working with data. “Don't let the data fool your intuitions," he said.

“Therein lies the rub” – combining the power of analytics with gut feel and years of experience.

“Gut feel is great for everyday problems,” said Silver. “But, it often leads us astray when we’re presented with complex streams of information. We still have “fight or flight” instincts and we react very quickly when we encounter a new stimulus. We can be blinded by the newest and nearest data point and miss the big picture. I’m urging people to slow down and think more slowly when they’re working with complex business questions.”

Which is why, according to Silver, explaining the past is not the same thing as predicting the future. This depends on two things: 1) whether an organization is in a data-rich or a data-poor environment; and, 2) the frequency of something occurring.

“In baseball, there are games every day, every season, every year. In [American] politics, you have one election – one data point – every four years. That makes it harder to judge your predictions. As an organization, you need to figure out where you are on that spectrum.”

Sometimes it’s as easy as measuring analytics maturity to understand how to move forward, and then align the entire organization around an analytics strategy.

Silver agrees, “You’re starting to see more interdisciplinary action within companies and I think that’s a very good development. The more you can get your hands dirty, the less divided you are as an organization.”

With all the noise around big data and analytics at IOD, it’s pretty apparent more organizations are on an analytics journey.

And, that’s a good signal.

For more information, watch a video interview of Silver at IOD.