Immersive Insights: Better data through AR

Q&A with Ben Resnick

Creative and Editorial Lead, Data and AI, IBM

Augmented reality (AR) and augmented intelligence systems such as Watson are breaking data outside the confines of a two-dimensional monitor and putting them into a three-dimensional visualization format. Big Data and Analytics Hub spoke with IBM AR designer Ben Resnick about what’s next for Immersive Insights and how data visualization will improve business intelligence for enterprise decision makers.

Big Data and Analytics Hub: Ben, you're one of the kind of key innovators and designers behind an entirely new type of data visualization tool which is set to change the way that data is consumed by presenting data in a 3D augmented reality (AR) format. What was the challenge you and your team were addressing when you developed Immersive Insights?

Ben Resnick: Today, data is super fragmented. Businesses have data in a lot of different silos. One of the biggest challenges that IBM is taking on is to make data simple. That’s the mission of IBM Analytics. That effort involves tools that do stuff like automatically aggregating and cleaning data. We saw an opportunity to build on top of all of that incredible technology. Immersive Insights will give IBM customers a better, quicker visual understanding of their business, built on a curated, unified view of their data. We saw that existing tools in this area fall short of their potential, so we set out to create the best possible experience, starting from first principles, and we’ve found that augmented reality and Watson are great ways to tackle that.

BDAH: How do augmented reality and Watson enhance the data experience?

BR: First, augmented reality lets you experience software in a more natural way. AR technology is still in its early days, so there are significant barriers to adoption, but the AR computing paradigm is not going to go away. It’s going to get bigger, and businesses are going to discover over time where it is valuable.  There’s no doubt in my mind that AR will be a major new source of value. For Immersive Insights, AR’s value is primarily about context and environment. AR allows you to compute with the entire world as your environment. Right now, office workers primarily experience their computers through these small 2D rectangles that sit on their desks — monitors. But in the near future, computers will sometimes be overlaid on top of your world, within your preferred environment. This will often be far less disruptive than sitting down, hunched over a monitor.

Context is equally important. When I sit down at my desk, I have a bunch of objects arranged in a very particular manner. I have my textbooks. I have my laptop, my pencils and my notebook. And there is this a very specific meaning to the way that all of those objects are arranged. I impose structure onto the world based on my mental model of how those objects relate. And this imposed model improves my ability to interpret and solve new problems. Think about how chefs arrange their kitchens for maximal efficiency. In the same way, we want to allow people to arrange the data that drives their businesses, spatially. It seems silly to constrain that data within a PC monitor screen.

Additionally, the depth perception and motion cues that users experience in AR improves their ability to interpret 3D plots.

With regards to Watson, it is obviously a super powerful technology. We have a really talented team in Germany that’s focused on creating a virtual assistant for Immersive Insights using Watson. AI, in general, is becoming more important, and emerging technologies, such as AR and Watson, are increasingly playing off each other and enhancing one another. The World Economic Forum has documented this phenomenon and refers to it as the “Fourth Industrial Revolution.” Among other things, it’s characterized by mutual reinforcement of value. It is the single most interesting trend that I’m aware of in technology today. There are these emerging technologies that seem very different, but when you combine them you get surprising power that you couldn't get with either one alone. And all of these technologies are developing so fast, so it's a really exciting time.

BDAH: Can you give some examples of how Immersive Insights works?

BR: Say you have a predictive model that might show you, for example, how much pollution there is likely to be tomorrow in Beijing. That model might be based off of 20 different variables, and on a dozen different pollution sensors spread throughout the city. And you might be a city official who is considering whether to shut down factories, reduce traffic, issue a pollution warning or make some other decision.

With Immersive Insights, you might start by use a dimensionality reduction technique such as PCA (principal component analysis). Dimensionality reduction would take a lot of different variables and reduce them into a 3D scatter plot that summarizes the important aspects of variance in your data. Then you could also generate a map of Beijing with the locations of the sensors and their readings superimposed. You could then walk around that map, similarly to how characters in Game of Thrones might walk around a giant map as they plan their next conquest. Then you could scrub through time to visualize predicted pollution at different moments and locations throughout the day. From there, you might start asking questions about all of this data, and Watson could respond. From that point, you might want to hone in on a particular aspect of the data. For example, you might walk over to a pollution sensor data point hovering over the southern Beijing part of the map, and then grab it. This would generate an info card with some additional context.  

So that’s an example of Immersive Insights coming into play when you want to weigh many different factors, in order to interpret the results of a predictive model – and to make some important decisions based on that exploration.

BDAH: And to your point, it seems like it would really help the data analyst create a more visceral or emotional connection between the data and its impact.

BR: For sure. We imagine that Immersive Insights is something that a CEO might use every morning. In the same way that you check the news, Immersive Insights could provide a high-level summary of your data. By compressing your business’s high-dimensional data, and creating this data space that surrounds you, we think we’re going to create a best-in-class solution. It's going to be a really fast way to see if something unusual is going on in your business. And then once you’ve conducted your daily exploration in Immersive Insights, you might use any number of different IBM tools to dig even deeper into your data.

BDAH: How easy is Immersive Insights to work with?

BR: Today this is a tool for data scientists and executives. You need to understand coding or, if you’re an executive, you’ll need to have the right data infrastructure in place. It's an advanced tool using emerging technologies. And right now we’re still in beta. There are large barriers to entry, which means that initially we think Immersive Insights will only be used for the highest value scenarios and most critical data. But these technologies are being adapted and developed so quickly, we could see massive adoption in the not-so-distant future.

BDAH: From what you’re telling me, it seems like this tool could make spreadsheets obsolete for evaluating and running your business. Are we at the beginning of the end of the spreadsheet?

BR: I don't think spreadsheets will go away anytime soon, but we are seeing a revolution in data. This revolution is being spurred by a variety of tools, many of which are made by IBM, so it could certainly be the beginning of the end. We are heading in a direction where AI will assist in an increasing array of use cases. And the infrastructures created by IBM Analytics, the Data Science Experience (DSX), and IBM Watson Data Platform are helping businesses to use their data more efficiently, organize it more accessibly, govern it more easily and certainly use it in a variety of contexts that go far beyond the spreadsheet. Immersive Insight’s role in all of that is to build off of that incredible foundation. It will be at the top of this sort of data technology pyramid, as part of a next generation data-consumption layer.

BDAH: You talk about DSX and its relation to Immersive Insights. Can you tell us more about it?

BR: Sure. So right now one of the things that we're working really hard on is making Immersive Insights as easy as possible to use and part of that is integrating it into a data scientist’s existing tools. We're working towards making it so that a data scientist can select their dataset within DSX, click a few buttons, and immediately be looking at their data in an AR headset. Then they can explore it, filter it, and export the results of their exploration back into DSX.

BDAH: Last year, during South by Southwest (SXSW), Immersive Insights drew a large crowd. What can the public expect to see at Think 2018?

BR: We've been working really hard on our cognitive assistant and filtering capabilities, and we're excited to reveal the details of those at IBM Think.

BDAH: There’s been a lot of talk about how to translate the AR/VR experience to the enterprise. What's viability for that and what do you think would be the most interesting applications for businesses today?

BR: Right now, we're not seeing too many production enterprise applications for AR/VR, but we think that's going to change in the next few years. Immersive Insights is on the leading edge of that. As AR/VR becomes more mainstream, I think we’ll see it increasingly viewed similarly to mobile: as a different computing context with unique pros and cons. So today, you might choose to create a mobile application, because it's for a business user on the go. Soon, similarly, you might choose to develop in VR because you want the enterprise user to be fully immersed in an experience. Or, you might choose AR for situations where a user’s environment contains cues that are important to their work. IBM is currently working on an AR/VR design language which will contain some guidelines for designers who are interested in learning more about this.

BDAH: There’s been a lot of talk about the rush to obtain data science skills and build the ladder to AI from data science and machine learning. For somebody like you on the cutting edge of all these developments, what are the things that you had to learn to get to a breakthrough spot in your skillset?

BR: Well, I’ve tried to take a broad approach and to stay up to date with all of the emerging technologies as much as possible. And that is certainly helpful. But if I had to recommend one thing, it would be to brush up on statistics.  Value is increasingly created by knowledge workers. A guy named Peter Drucker said that 60 years ago. He means that people are spending more and more time manipulating data, and less time doing physical labor. So statistics are becoming increasingly foundational. As data plays a larger role within organizations, just knowing a little bit of applied statistics is super helpful.

BDAH: Did you have a design background or any sort of visual arts experience?

BR: My background is in computer science and interactive multimedia environments. So working on Immersive Insights is pretty much a dream job.

Try out Immersive Insights for yourself at IBM Think in Las Vegas’s Mandalay Bay 19-22 March. A demo will be featured throughout the week in the Cloud and Data Campus’s demo area at ped 609. The demo will show how Immersive Insights can be used in conjunction with Aginity, H20 and DSX to help predict the next great baseball pitcher.

Register here.