Top 5 trends that are making life easier for data professionals
It’s an exciting time to be a data professional. New roles are being created while traditional roles are evolving. Data professionals need to address current challenges such as capturing data streams for insight and building more intelligent applications that can respond and learn. Data professionals are also rightfully demanding new ways to explore, discover and interact with all data types from structured, transactional data to voice and image data. One of the objectives is to spot unexpected trends that might tell a new story or present a fresh approach to an old problem.
Some, but not all, of the data professional players are data scientists, data engineers, developers and designers. The roles aren’t strictly defined and there is some overlap, which is actually the perfect recipe for innovation and collaboration. We could, however, draw a few distinctions.
Data scientists represent an evolution of the business or data analyst role. The formal training is similar, with a solid foundation typically in computer science and applications, modeling, statistics, analytics and math. What sets the data scientist apart is strong business acumen, coupled with the ability to communicate findings to both business and IT leaders in a way that can influence how an organization approaches a business challenge.
Data engineers design, build and manage the data infrastructure. They develop the architecture that helps analyze and process data in the way that's most appropriate for the organization, while making sure those systems are performing smoothly. As a result, they work closely with the developers. The application developer creates, tests and programs applications software for computers.
At the end of the priority list comes the application designer. Applications fueled by data need to be usable by everyone; it’s said that the app should be so well designed a cat could use it.
As business requirements move us into a truly insight-driven economy, data professionals must alter their work methods accordingly. Fortunately, help is available. They can get some well-earned free time back in their days or more likely, knowing data professionals, invent the next intelligent application. Here are a few trends that are making life more fun for data professionals.
- Apache Spark: Spark is a big data processing framework built for speed, ease of use and sophisticated analytics that enables applications in Hadoop clusters, or stand-alone apps, to run 100 times faster in memory and 10 times faster when running on disk. According to Gartner, by 2018, 30 percent of near-real-time data integration and data management use cases will be supported by stacks that include Apache Spark. Spark enables data scientists, developers and data engineers to work together to access all data, build analytic models quickly, iterate quickly in a unified programming model and deploy those analytics everywhere.
- Skill reuse: Emerging technologies such as Apache Spark and the Open Data Platform enable data professionals to use existing skills such as SQL, Java, Scala or Python. Existing models built with R can be deployed, scored and updated on platforms such as IBM Streams. Systems expertise for file system management in Hadoop Distributed File System and IBM GPFS will scale across Hadoop and Spark. These are just a few examples.
- Education: The Big Data University and leading-edge education providers such as Galvanize are constantly updating their curriculums to help data professionals improve collaboration. Find massive open online courses (MOOCs) at Big Data University for Spark and Hadoop.
- Cloud: It lowers the barrier to productivity. IBM recently opened an Apache Spark Service on IBM Bluemix. Bluemix has a host of other integrated services for data professionals such as streaming analytics, Hadoop and databases.
- Community innovation: There are over 500 active contributions to Spark. IBM opened the first-of-a-kind Spark Technology Center in the heart of San Francisco. The community realizes we are stronger together than individually. Data is a team sport.
Join us by registering for the webcast “Build smarter applications fueled by data with IBM and Apache Spark” to explore these in more depth. You’ll see an application built in ten days involving all of these trends.
And to experience deeper insights from data-driven analytics, explore Spark and other tools and platforms in the comprehensive IBM Analytics solution family.