Open source is opening new frontiers in the insight economy

Big Data Evangelist, IBM

Openness unlocks creativity. When people freely share their ideas and work, humans as a whole benefit. Being able to tap into a deepening pool of shared assets encourages collaboration and innovation. New industries spring up, and old industries receive a healthy jolt of fresh energy.

Innovation through openness the insight economy, openness means many things but is principally expressed in open-source communities, especially Hadoop. This and similar industry initiatives have created big data analytics platforms that have seen wide adoption and that power a vast range of commercial applications.

Being the chief platform for many big data analytics initiatives, Hadoop sustains entire industry ecosystems of innovation. As Hema Reddy has discussed, Hadoop provides a robust enterprise framework to support “rapid innovation driven by millions of developers around the world and by companies working collaboratively to address the growing needs of dynamic, fast-paced technological growth.” Indeed, years after its inception, Hadoop continues to provide an umbrella under which innovative open-source codebases—among them HDFS, MapReduce, Spark, YARN, HBase, Hive, Ambari and Pig—have gained adoption in myriad applications.

Open data as a public good

Regardless of the platforms used to develop big data analytics, open data is a key resource powering the insight economy—a public good for national economic development, as the McKinsey Global Institute has pointed out in a groundbreaking research report. Thus the public at large benefits when open data is made freely available from diverse information sources without the exaction of royalties, licenses or other fees. Though such availability might adversely affect some organizations’ revenue streams, the macroeconomic benefits of open data might well outweigh the benefits of keeping information closed and proprietary, locked behind a “paywall.”

In its report, McKinsey identified some important structural innovations that—if implemented widely—would help bring about understanding of the economic potential of open data while protecting commercial interests from piracy and guarding individuals from intrusion into their private lives. Chief among McKinsey’s proposals were accelerated adoption of information-sharing standards, implementation of regulatory environments for protection of intellectual property and creation of comprehensive privacy safeguards.

Equally important for national economic development is the development of transparently democratic institutions, which also rely strongly on open data. Accordingly, vibrant democratic societies around the world are taking the lead on open data initiatives, many established and promoted by transnational groups such as the Open Data Institute, the Open Government Partnership, the Data Transparency Coalition, the UN Public Administration Programme and the World Bank.

Open expertise for economic development

Like open data, open expertise is integral to economic development in the era of big data analytics. Indeed, innovations are springing from open expertise marketplaces, such as Kaggle, and a new generation of data scientists is acquiring skills and knowledge thanks to massively open online courses such as those offered by the Big Data University. Moreover, an increasing number of data-driven analytics application developers are making a living by crowdsourcing open-source expertise.

Attend IBM Insight 2015 in Las Vegas on October 26–29 to learn what data science, a primary driver of the insight economy, has to offer your career.