Hadoop Summit 2015, Day 3: Passing the torch to Apache Spark

Big Data Evangelist, IBM

I’ve logged some serious conference mileage in my life. I’ve never stopped to count the number of industry conferences, expositions and seminars I’ve attended during the 30 years of my IT career. But I reckon it’s in the hundreds. Just thinking about it exhausts me. But I still love conferences because there’s no better opportunity to connect with people in the industry and to gauge whether they share my passions for the latest technologies that consume our careers.

Hadoop Summit 2015 was my fourth one in the past five years, and the attendees have always been just as passionate about the technology as I am. But something was different about this year’s summit, as compared with previous years. Apache Hadoop is no longer the next big thing in big data analytics. And nobody at the event tried to pretend that it was. That torch has clearly passed to Apache Spark, which was as promising and as unproven as Hadoop had been when I attended my first Hadoop Summit in 2011. waning of Hadoop’s hip factor didn’t seem to matter to attendees in any way. Enthusiasms were high, business is good and people are doing exciting work. Every other presentation at the summit seemed to end with the bullet point “we’re hiring,” and the jobs board near registration was densely festooned with hot opportunities. Attendees were keenly focused on the content of all the sessions I attended, and there seemed to be very little attrition in attendance from one day to the next.

Clearly, the event is an annual forum that its community depends on for recruiting, professional development, marketing, sales and other important reasons. As this same community of big data analytics professionals shifts its attention to next week’s Spark Summit, we need to remind ourselves that the newer technology is still unproven commercially. Unlike Hadoop, few enterprises have bet the business on Spark. And there are still precious few Spark-specializing vendors, systems integrators and consultants in the market.

I’m sure that situation will change as Spark starts to ramp up toward broad enterprise adoption. I suppose that those of us attending Spark Summit in San Francisco will get the inside scoop on that progress. Register to attend Spark Summit, June 15–17, 2015, in San Francisco, California. And please join data scientists and other big data professionals on June 15, 2015 at Galvanize, San Francisco for a Spark community event. Hear how IBM and Spark are changing data science and propelling the insight economy. Sign up to attend in person, or watch the livestream, and to receive a reminder notification on the day of the event.