What Koby's tea leaves foretell for big data in 2014

Big Data Evangelist, IBM

Year-end predictions are like yule logs: everyone wants to throw their own on the communal fire to see if it makes a joyous crackle. Even if it's only for one's own edification, it's good to collect your thoughts on what might trend, what might end and what the main disruptors will be in the fast-changing information technology (IT) arena.

Predictions are what people expect IT evangelists to spin like mental gossamer, and I've always obliged. For several years running, I've been producing year-end predictions for what we now think of as the big data industry; see here for last year's predictions (my first as an IBMer) and here, here and here for the previous year's (my last as an industry analyst).

If you delve into the aforementioned links, you'll notice consistency from year to year, but with continual shifts as the industry evolves and my radar crowds with interesting new trends. Big data industry predictions are a delicate art; they need to reflect original thinking but not be too blue-sky. They need to have integrity and be grounded in the unique viewpoint of a specific observer. And they need a practical utility that guides IT professionals in realigning their own planning frameworks.

When spinning predictions for the coming year, one needs to avoid being too obvious, too broad, too specific, too self-serving and too vendor-centric. For example, there's no great value in being told yet again that big data will stay hot, that it will evolve into the cloud or that data scientists will still be sexy, smart individuals in great demand. But, if you're looking for useful trends from a planning standpoint, it's also pointless to fixate on whether particular solution providers will do well, ship particular products or make specific acquisitions in the coming year. And if you're affiliated with a particular solution provider (such as yours truly), you might unconsciously align your predictions with your employer's party line, unless you're diligent at maintaining the proverbial 30,000-foot view.

Looking back at the year gone by, I realize that I've made many big data industry predictions in the context of my blogging and other thought leadership activities. Here now, with self-authored sources indicated, are my big data industry predictions for 2014:

  • Big data computation will become fundamental to scientific progress. In 2014, big data will become a backbone of modern science, and empirical investigations will become computational in nature, leveraging big teams, big tools and big data to tackle big problems that cannot be addressed as effectively at smaller scales. Source: "Big Science?"
  • Open reference big data will drive macroeconomic growth. In the coming year, more nations will retool their legal and regulatory systems to encourage development and sharing of open reference data. This trend will be fundamental to unlocking the promise of big data as shared resource for economic innovation and the public good. Source: "Open data? The macroeconomic multiplier effect"
  • Real-world experimentation will revolutionize business. Over the next 12 months, more businesses will align their operating models around continuous real-world experimentation driven by advanced analytics, next best action platforms and big data. This trend will accelerate the evolution of data science into a 24x7 operational function. Source: "Big data demands nonstop experimentation"
  • Data scientists will become key convergence developers. As we push into the new year, the convergence of big data, cloud computing, social media, mobility and Internet of Things will pick up speed. Data scientists as principal developers of the analytic models and business rules that drive this conergence. Source: "Data Scientists: Key Programmers in the Convergence of Big Data, Cloud, Streaming and Internet of Things"
  • Cognitive computing will pervade all analytics. In 2014, more big data, advanced analytics and business intelligence platforms will adopt cognitive-computing capabilities to automate sense-finding, natural language processing, decision-automation and semantic search functions against all data and media types. Source: "Cognitive Computing: Relevant at all Speeds, Scales and Scopes of Thought"
  • Mathematical insights will deliver deepest value. Going forward, many of the most disruptive big data initiatives will leverage breakthrough mathematical insights encoded as powerful new algorithms. This trend will counterbalance the increasing tendency of data scientists to rely on brute force processing, cheap computation and larger data sets to unlock analytical insights. Source: "Big Data and the Power of Mathematical Breakthroughs"
  • Machine learning will boost data scientist productivity. From now on, data scientists will adopt machine learning as a productivity tool. Data scientists will gladly offload many cognitive processes to automated systems where there just aren’t enough flesh-and-blood humans to exercise their highly evolved brains on various analytics tasks. Source: "James Kobielus: Big Data, Cognitive Computing and Future of Product"
  • Big data will evolve into a general purpose development platform. As we move into the middle years of this decade, broader adoption of YARN will transform Hadoop into a general purpose, distributed programming and execution layer for a wider range of workloads than MapReduce has been heretofore capable of supporting. YARN will also begin to pervade a wide range of big-data platforms beyond Hadoop. Source: "YARN unwinds MapReduce's grip on Hadoop"
  • Hadoop appliances will become core building blocks of the big data cloud. Continuing the trend from 2013, big data will evolve into a cloud ecosystem. Appliances will become the dominant approach for enterprises to deploy Hadoop and other emerging big data approaches in private clouds. Source: "Hadoop Appliances: Key Building Blocks of the Big Data Cloud"
  • NoSQL databases will find their core niche applications. In 2014, the NoSQL market will stop trying to position their platforms as direct competitors with relational databases for data warehousing and online transaction processing applications. Instead, NoSQL vendors will increasingly position their platforms into specific niches where eventual consistency is no big handicap. Examples of such niches include tactical, horizontally scalable repositories for specific content management, document management, text analytics, semantic analysis, graph analysis and other multistructured data applications). Source: "The NoSQL conundrum: Lagged veracity and the double-edged promise of eventual consistency"
  • Streaming media will drive big-data evolution. Through the decade's end, streaming media will deepen its footprint into entertainment, advertising, marketing, education, music, community and practically every other aspect of online culture. This trend will push big-data architectures to new levels of scaling along all of the "3 Vs." Source: "The Emergence of Big Media: Evolving and Dwarfing Today’s Big Data Platforms"
  • Big data privacy will become high enterprise priority. With the surveillance controversies of 2013 still roiling the global community, the new year will see businesses everywhere put privacy high up on their big data planning agenda in the coming year. Organizations will pay specific attention to the uses and abuses of identity resolution and related technologies. Source: "Big Identity’s Double-Edged Sword: Wielding It Responsibly"

Looking over my previous years' predictions, I can see that I was spot-on with most of these trends. You should construe these latest predictions as supplementing and extending the prior years' outlooks. They encompass my outlook for 2014, but point to how big data will continue to evolve, in myriad directions, through the end of this decade.