Small Data + Big Data = Major Awakening in U.S. Election
While we eagerly await the first post-election “victory lap” article by Nate Silver – who correctly predicted the outcome of all 50 states in the United State Presidential election Tuesday – I want to share with you several of the top articles that address the role of data and analytics in this election.
An interesting dichotomy jumped out at me in this roundup: big data vs. small data. Nate Silver’s analysis is based in simplicity. He himself said, “[My model is] not really that complicated, but people treat it like it’s Galileo or something really heretical.” In his book, The Signal and the Noise: Why So Many Predictions Fail, but Some Don’t, he said, “…our predictions may be more prone to failure in the era of Big Data. As there is an exponential increase in the amount of available information, there is likewise an exponential increase in the number of hypotheses to investigate. . . . Most of the data is just noise.”
On the other hand, the Obama campaign apparently thrived on big data, using teams of analysts huddled in a secluded, windowless office, poring over every data element and correlation to find a sliver of an advantage they could exploit to gain votes.
In my daily trolling of social media, I often see people adamantly state “Let’s stop with the ‘big data’ moniker! Data is data!” I wouldn’t say that. There is definitely a difference between data you manage with Excel and data streaming through interconnected sensors and systems.
What I would say is “analysis is analysis.” Be smart about it.
"How Obama's data crunchers helped him win," by Michael Scherer, Time, Nov. 7
A massive effort to find, analyze and use data helped President Obama raise $1 billion, remake TV ads, and increase the effectiveness and reach of "feet on the street" campaigners. Just like most organizations, they had to tackle the issue of data silos. In the end, one Obama campaign official proclaimed, "in politics, the era of big data has arrived.
"Election Result Proves a Victory for Pollsters and Other Data Devotees," by Michael Cooper, The New York Times, Nov. 7
Election predictions pitted the traditional “Spidey sense” of crusty old political wonks against newfangled analytical science and statistics. Cooper provides an excellent roundup of pre-election predictions by pundits and pollsters, many of whom were left scratching their head and wondering where they went wrong while others smirked and cried, “I told you so!”
"This is about the triumph of machines and software over gut instinct." Lyons looks forward to the day when intelligent machines are embraced for their power and accuracy, unlike the backlash that has occurred over the last couple of decades when computers like Deep Blue and Watson have bettered their human counterparts.
“Triumph of the Nerds: Nate Silver Wins in 50 States,” by Chris Taylor, Mashable, Nov. 7
Silver might cringe at the opening-paragraph statement “here is the absolute, undoubted winner of this election: Nate Silver and his running mate, big data,” but otherwise Taylor does a fine job of sticking up for analytic geekdom.
“Math and Discipline—Why Nate Silver’s Accuracy Isn’t About 'Big Data',” by Kent Anderson, The Scholarly Kitchen, Nov. 8
The Scholarly Kitchen is the Twitter account for the Society for Scholarly Publishing, which explains its carefully measured take on Nate Silver's phenomenal success in predicting the U.S. Presidential race over the political pundits, whose standard is "ratings, not accuracy." The post explains the relative "small data" Silver used and why his attention to analysis, not emotion, aided him.
Hear directly from Nate Silver
If you would like to hear Nate Silver talking about his theories and approach to analysis and predictions, watch this interview from the recent Information On Demand 2012 conference.