Spark ignites conversation around building smarter apps

Digital and Social Marketing Strategist, IBM

When Apache Spark experts converged in the informative and thought-provoking discussion “Spark: Building smarter apps,” the CrowdChat—the third in a series on Spark—proved to be a stimulating hour for all the data professionals who contributed, as well as for their Twitter followers. Indeed, the chat’s 3.2 million Twitter impressions and 2,122 views were a clear testament to its broad reach and level of engagement.

If you don’t have the time to read the entire chat transcript right now, I’ve excerpted some of the high points here. For each excerpt, I’ve included the question that triggered the thought. But don’t miss out on what the full chat has to offer: These excerpts are merely choice snapshots of what some very intelligent people had to say about building smarter apps on Spark and on open data platforms generally.

What real-world challenges are driving developments of smarter apps?

“There are a number of trends: a move to real-time analytics is starting, there are new devices generating new streams of voluminous data, there are computing devices in the hands of even relatively impoverished people, raw network speeds.”—Andrew C. Oliver
“Real-world challenges requiring smarter apps are simply the volume and velocity of data one should take into account, which is beyond what humans can correctly process. Also, for many real-world problems, algorithms have improved to outperform human experts.”—David Talby

How can simple APIs improve Spark developer productivity?

“Standard APIs are the key to simplicity on the development side. With standards, Spark apps only need to be written to those APIs in order to execute on any platform that runs Spark engines and can, via Spark SQL, tap into HDFS data.”—James Kobielus
“Helps explore previously untapped data like the streaming API.”—Kimberly Madia

What mix of skills and knowledge enables data scientists to build smarter Spark apps?

“I don't fundamentally believe in data scientists as people imagine them. There are good mathematicians who write bad python. There are bad mathematicians with a distributed computing background and there are business SMEs.”—Andrew C. Oliver
“To build smarter Spark apps, data scientists need to build their models from data in ‘lakes’ that include all relevant sources, use appropriate algorithms from common libraries, and establish tight governance over data and models.”—James Kobielus

How can the Spark community best pool its brainpower to produce smarter apps?

“Big Data University attracts the best minds with its social badge program. [P]ass the course, get a badge, build your cred.”—Kimberly Madia
“Make the most from the 500 active contributions to Spark. Work to build more centers like IBM's FOAK Spark Technology Center at San Francisco.”—Mahanth CH

For more insight surrounding Spark, read the full transcript of the CrowdChat “Spark: Building smarter apps.” After that, keep on exploring the power of Spark.

Also, don’t forget to register for the IBM Insight 2015 conference, where you can become even better acquainted with Spark.