Innovative applications abound at the first Spark hackathon

Content Marketing Manager, Emerging Markets, IBM

IBM and the Hack/Reduce community ran their first ever Apache Spark Hackathon in Cambridge, Massachusetts from May 28–30, 2015. Spark generated a lot of excitement in the Boston data science community. A mix of students, academics and professionals came together for the same reason—to learn Spark and use it to build applications.

“The hackathon was an excellent experience,” according to Team WarrenBuffet, which consisted of Eric Gieseke, Anindita M. and Sriram Moorthy. “The IBM team did a great job organizing, answering questions, setting up Slack [messaging] for collaboration and configuring the IBM Cloud for our application hosting. Our team learned a lot about Scala, Spark and the [Spark scalable machine learning library] MLlib. With these powerful technologies, we were able to create a stock-sentiment analyzer in the course of a few days. The technology is extremely powerful. We are looking forward to the next hackathon.”

Academic reach

Sri Krishnamurthy, Chartered Financial Analyst (CFA) and Certified Analytics Professional (CAP), teaches an advanced graduate-level data science course in the information systems program at Northeastern University. A few of his students were participants in the hackathon, and thanks to personal connections, Krishnamurthy decided to not only participate, but also to get his entire class involved.

On presentation day, the team at Hack/Reduce set up a Google hangout to stream the remote presentations to Krishnamurthy’s classroom at Northeastern. The streamed presentations gave Krishnamurthy the chance to get his final presentation in and give his students the chance to see all the innovative Spark Hackathon submissions. Krishnamurthy’s involvement with the data science community and his incorporation of his graduate program was well received. Together, we hope to continue to incorporate the university with our developer presence, not only in Boston but globally.

Hackathon winners

Team Explorer’s WebSpark took first place. David Wang stole the show with WebSpark, a web front server that enables Spark cloud instances to serve as web end points. “WebSpark is transparent to the web browser and HTTP clients,” Wang said. “Web clients receive standard HTTP responses, which are generated by Spark cloud instances. Otherwise, these Spark cloud instances do not normally accept inbound HTTP connections.” Wang’s solution enables URLs to transparently map to web client requests, eliminating the need for task queue support for web clients and Spark instances. Wang’s code can be viewed on the GitHub website.

Team WarrenBuffet took second place for its stock-movement forecaster app, called Invest for Me, which predicts stock-price movement based on social sentiments by monitoring Twitter feeds that capture word frequency associated with a stock. The model utilizes the Spark MLlib to feed in the stock price changes and word-frequency data. This process enables the model to continuously update and apply real-time data to predict stock price. The final component is a prediction and a recommendation to either buy, sell or hold a stock. Check out the application’s technology architecture, and view the full code.


The next hackathon

Visit Spark Summit in San Francisco, California, June 12–14, 2015, where a second hackathon will take place. Get more details and register today.

Be sure to check out the variety of Spark enablement resources, including information surrounding upcoming Spark meetups, hackathons and courses.