Making Data Simple: From 2D to 3D -- augmented reality data visualization

Making Data Simple: From 2D to 3D -- Augmented Reality Data Visualization


Augmented reality will change the way we experience data. But, what will this new reality look and feel like, and how can we take advantage of it to make better decisions? Tune in to this episode of Making Data Simple to learn about augmented reality data visualization from Ben Resnick and Alfredo Ruiz, members of the IBM Immersive Insights team.   


00.10 Connect with Al Martin on Twitter (@amartin_v) and LinkedIn (linkedin.com/in/al-martin-ku)

00.25 Connect with Benjamin Resnick on LinkedIn (linkedin.com/in/benjamin-resnick-7b602b7b), view his online profile (http://benjaminresnick.com/), see his medium account (https://medium.com/@benjaminresnick) or email him at resnick@us.ibm.com

00.45 Connect with Alfredo Ruiz on Twitter (@alfredoruizc), LinkedIn (linkedin.com/in/alfredoruizc), view his profile (http://www.aruizdesign.com/), see his medium account (https://medium.com/@alfredo.ruiz/) or email him at alfredo.ruiz@us.ibm.com

01.40 Learn more about the Blue Unicorn Initiative at http://www.blueunicorn.nl/

02.30 Connect with Rob Thomas on Twitter (@robdthomas) and LinkedIn (linkedin.com/in/robertdthomas) and read more of his work on his blog https://www.robdthomas.com/

02.40 Connect with Jay Griffin in LinkedIn (https://www.linkedin.com/in/james-griffin-43094711)

02.45 Watch Elon Musks video at http://www.businessinsider.com/elon-musk-first-principles-2015-1

03.50 See Visualizing High Dimensional Data in Augmented Reality at http://bit.ly/2xkdZpS

04.20 See image of Daenerys looking at the map of Westeros at http://bit.ly/2yLdY1k

21.20 See Benjamin and Alfredo's award nominations here:

  • Information is Beautiful Award 
  • Driven by Design:

22.10 Find The Fourth Transformation by Robert Scoble at http://amzn.to/2iV2JtR

22.30 Find Ready Player One by Ernest Cline at http://amzn.to/2xlgjwO

23.00 Find Made in America by Sam Walton at http://amzn.to/2gzl5A2

Hungry for more? Check out our previous podcast episodes of Making Data Simple:



Moderator: Al Martin

October 20, 2017

10:06 am CT

Al Martin:            Hey folks.  Welcome back to the series Making Data Simple.  This is Al Martin speaking.  I hope this message meets with you well.  This morning, I will be interviewing two people on the topic of Augmented Reality Data Visualization.  It's kind of a mouth full.  We're going in deep now I think at this point.  

                              And I've got two experts with me today like I said, Ben, Alfredo.  Ben, why don't we start with you?  Why don't you introduce yourself and then I'll turn it over very quickly to Alfredo? 

Benjamin Resnick:    Sure.  I'm Researcher and Front End Developer in IBM's Design Department.  I spent some time working on IBM's Graph Database and some time as a Quantitative User Researcher for IBM Analytics and now I work with Alfredo and Jay Griffin full time on IBM Immersive Insights.

Al Martin:            Fantastic.  Alfredo? 

Alfredo Ruiz:       Sure.  Hello I'm Alfredo Ruiz and I'm the Design leader of Immersive insights and we've been working on this probably for like a year and a half and working with 3d visualizations and now starting to use augmented reality. 

Al Martin:            Awesome.  So you guys are, like I said, you're delving into new frontiers here.  So we've targeted big data.  We've talked about making data simple.  That's obviously the name of the podcast but now augmented reality, I think most people listening think about augmented reality in terms of I don't know, like the Samsung glasses that you put on and then you start playing games. 

                              You guys are applying this to data.  Can you just talk about what it is, what got you here, why it's important? 

Benjamin Resnick:    Yes sure.  So it's been a long road.  Alfredo started this project through Blue Unicorn Initiative. 

Alfredo Ruiz:       Yes so we started this project, as I mentioned like a year and a half and I joined IBM two years - a little bit more than two years ago - and the first thing that I started - my background is industrial science.  And then I started doing (unintelligible) and I've been working on some explorations with data and I have big passion for 3-D objects. 

                              So we started exploring this, how we can visualize information in a beautiful way.  From there, we become this advanced concept group exploring this type of idea and later (Rob Thomas) created initiative called Blue Unicorn. 

                              So we partnered the design team, we partnered with some development folks.  Jay Griffin was our developer who also started exploring these ideas and become Immersive Insights. 

Benjamin Resnick:    So part of how we thought about is what is the best way to understand data if you start from first principle, there's really great interview where Elon Musk talks about like, how he thought about designing a rocket ship and what's, if you break it down to the raw material not what's today but what is theoretically possible, what's the best possible way to do this? 

                              We sort of start to think about data visualization from that perspective and with the five senses that we have, what's the best possible way to get the most useful information and the best signal to noise ratio of useful information into the analyst’s brain and we think that 3D is a big part of that and using the entire natural world around that as opposed to just the T.V. screen. 

Al Martin:            So I went out there and you've got a great video on YouTubecan you describe that experience for our, the listeners here?  I mean, remember they're on there.  They'll go after YouTube I'm sure of it.  Can you describe what it feels like and what you're trying to accomplish within that video by example? 

Benjamin Resnick:    Yes sure so for example, with Immersive Insights, you're looking at a map.  

                              We've managed to make to it so you can look at a map of the New York City and you can actually scale this map to fill a room sort of in the same way like, if you watch Game of Thrones, Daenerys Targaryen is looking at this big map of Westerosand that's how generals look out at their battle fields but we think that executives who are figuring out their supply chain might actually enjoy having a map at the world on the you know, the table in front of them and analyzing the factories are having issues, that type of thing like the digital experience of having the world laying out on that table right in front of while you're conducting that meeting.  

                              Likewise for the other types of analyses we're doing which are more technical and more data scientist oriented like Immersive Insights, what I have experienced personally as a user is this idea of being surrounded by data. 

                              So if I'm thinking about a particular dataset, it's on my mind constantly and I'm thinking about it in new ways but if I'm wearing a pair of augmented reality glasses and looking at the data around me in the space that I live in, it becomes this physical manifestation of my thought process and of my analysis.

Al Martin:            I think you answered one of my questions.  My question was is are we going to be looking at a C Suite executive at some point that's going to have a helmet on and they're going to be like, reaching out into space you know, as you're, as you're walking past their office?  I mean, you've got to wear a helmet or glasses to immense yourself into it? 

Alfredo Ruiz:       I think that's, we're getting there.  If you see like a, you know, all the big players in tech, they're putting out a lot of money in augmented reality.  A good example was like, the new iPhone, right, that they're adding augmented reality. 

                              So I think it's going to be a evolution and we can you know, go back and see when we start with the punch cards and then we move to the laptops then to mobile where like, everybody's using like, a smartphone, right? 

                              And I think that's part of the changes we're going to start experiencing now with AR  I think we are, we're coming to this phase of where we're emerging from a model from these things that we're having in our hand to now having something in our, in our face. 

                              It's some glasses where we can you know, get information and experience the world in a very different way and I think we're moving towards with that.  

Al Martin:            You know, as a side note, when I think about augmented reality, I envision a day where you know, like we're going to have the super bowl, you know, the big game in the U.S. for football where we're going to have all these fans set up in these different seats where I'm going to be at home and I'm going to be on the 50 yard line and I'm watching this thing like I'm sitting there.  

                              I think that day is very near.  Anyway, so you mentioned immersive insights.  Can you define that for me? 

Benjamin Resnick: Yes.  So what, immersive insights is we're focusing right now into, with data scientists.  The idea is that we're helping them explore information in a different way by enhancing these virtual tools, right? 

                              When we talk about AR, the big thing with this is that you can actually figure to real environment as well.  If we compare to virtual reality, you're basically immersed in a different world.  

Al Martin:            So it's not… 

Alfredo Ruiz:       Yes.  So with any AR, you can actually see your environment, what would your real tools say for example, right now we're using data science experience so that data scientists can use data science experience, work with their data, analyze it, and then go a little bit deeper with immersive insights putting on a headset and they can now actually start walking around their data and start you know, seeing this relationships and (unintelligible) maybe with a regular (unintelligible) that might take like, some hours and with these exploration in this immersion, you can start seeing patterns and relationships in just seconds.  

Al Martin:            Let me ask you another question.  So yesterday, just yesterday I did a keynote and I talked about you know, Making Data Simple and all the different elements of data from hybrid data management to unified governance to analytics and visualization and when I got to visualization, kind of what you talking about here, I talk about four different elements. 

                              Planning analytics, that obviously answers what is your plans, descriptive analytics, it answers what happens, diagnostic analytics is where we would bring Watson in that is why did it happen and then finally predictive analytics or what's going to happen next you know, and the product that we have at (unintelligible).

                               So do you envision like a (unintelligible) form of augmented reality to look at this data or do you see augmented reality encapsulating all those four elements that I just described? 

Benjamin Resnick:    I'm going to actually say that it's just another method of conducting those methods you described and as enterprises, I've actually heard it described another way which is this journey from descriptive analytics and to predictive and then to the automated decision making so that's sort of another framework for thinking about (unintelligible) curve where you have this (unintelligible) understanding of your data, second of all, suggestions about what you should do with that data and finally cognitive systems that can act on that data without human intervention or perhaps just with approval.   

                              And I think it's likely that we'll see systems like Immersive Insights that allow you to understand what those cognitive systems are doing, what that analytic system is thinking.  

                              Just some sort of 3D representation within your world.  This information that's you know, relevant to what your tact is, happens like, you get that information instantly so like, when we talk about the transition from mobile to AR, we're talking about getting data faster so you don't even have to pull out your mobile phone.

                              The moment that important alert occurs about your stock crashing, about a disruption to your customers, you can get that in your headset that moment and complete information. 

                              So the complete information, like, when we're talking about the Instacart data set which is the data set of people and information on all of the food that they've purchased, that's like sort of describing who that, who that user is to their Instacart product team. 

                              With immersive insights and dimensionality reduction techniques, you can actually visualize all of those users in 3D space so one of the things we want to do is to be able to summarize all of the important information to your business that a predictive analytics took might give you different ways of diving into and to take that high level summary and act on it. 

                              So that might have been a little rambling but…

Al Martin:            No, no, that's good.  I mean, actually this is fascinating to me.  

                              Here's the thing and this may be a question that you kind of already answered but I'd like to kind of hit it again and I think it'd be good for listeners, at least it would be good for me so I'm assuming it's going to be good for listeners and that is look, if I'm in a C Suite, or it doesn't matter, any persona out there, and I just described to you a minute ago you know, visualization technologies that already exist today, what is going to cause me to say the current visualization or you know, the current technologies just don't get it done and I'm going to put on that the glasses, that headset, whatever, to get into 3D.  

                              What would promote me to do so? 

Alfredo Ruiz:       Well I think it's the, like terminology that we've been like, talking about is for example, when you're seeing a picture of a forest, right?  You see, looks beautiful like you can see a few trees and all of that and how different is when you actually arrive to that like, you can actually see much more details, you can walk around it, and experience it in a very different way. 

                              And that's the way we feel with augmented reality.  With Immersive Insights, you can actually walk around the data and sort of see these different patterns and we are, we're used to see you know, the one or two dimensions, right? 

                              So our brain, it's better for us to understand information when we see it in this way so I think that's a big, it's a great benefit for having this type of tool then and you know, when you're like, (unintelligible) different type of users and you can see information and this patterns this way.  I think it's going to be a game changer. 

Al Martin:            I mean, is that to say that you know, it's kind of like anything I mean, you can look at, there's many ways to get data, pull data, see data but because you know, just you know, we're used to the 3D images, you think it's going to be more organic by which we'll get more out of it. 

                              It will be easier to visualize hence the name visualization and be able to drive data insight with augmented reality, the 3D images that you're talking about? 

Benjamin Resnick:    Yes.  And I think there's a couple different aspects to why we need this project, why we see this becoming a big piece of the future.  One instance, we deeply that AR is the best way to accomplish a wide variety of computing tasks.

                              So to give a concrete example, you have a desk probably in front of you and that desk has a jar of pencils, maybe it has a stack of books and it has these items that are arranged in a very specific way that matches your, this mental model that you have of the world that you then exert on the world. 

                               You control it and you can shape it in this way that makes sense to you and helps you interpret and understand. Right now with computers we're existing in this world of 2D projections where you stare at the screen, you manipulate text documents and web pages and you're just looking at two dimensions.

                               But there's three dimensions in the world and like, one of the best things we have is depth perception like, there's so much stuff our brains doing behind the scenes that is just subconscious that all these environmental queues we pick up on.  

                              So you could have the graph of say, real time usage information and you can just put it on your bookshelf and where these, where they assume that you're wearing glasses everywhere you go in six years and this is just the next computing platform. 

                              Then when you walk into your office, immersive insights has generated this graph on your bookshelf and this place where exactly where you want it showing real time actionable information about what your customers are up to today and you don't have to dig around in a 2D screen and a folder to find that so that's one aspect of it.

Al Martin:            So you talked about Instacart you know, the Instacart which is a good example.  Are there any other examples that you'd mentioned or is that probably the best example that the audience can put their mind around? 

Alfredo Ruiz:       Well I think like, there's, we have like, a few posts and the like, the medium can definitely take a look at our medium accounts and see what we've been writing.  We've been exploring like, different fields like finance and other industries as well so I think we have a good range of opportunity here so yes. 

Benjamin Resnick:    I'd say so my background is in quantitative user research.  It's very easy for me to think about all of the up source of a business and for me, part of being on the scene is using this tool to address questions I'm interested in about how customers behave. 

                              So in these cases where your mapping out all the users for your businesses in 3D space and trying to understand the relationships that come naturally to me but as it seems, certainly if there's any data scientists out there listening to this, if you want to get data users in the next month or so, we would love for you to reach out to us and to hear about your ideas for what kind of visualizations would be useful to use. 

Al Martin:            That's awesome.  So two part question is we're kind of starting to wrap up a little bit.  So is this technology available today or when do you think it's going to be available and I guess further what do you think the future of this technology is? 

Alfredo Ruiz:       So yes, we're starting to work on some integrations within data science experience so that's our next stage for our project.  So for we have done a demos we represent this (unintelligible).  We also contracted data and in many other places. 

                              (Rob Thomas)for example did a, he was in the (unintelligible) in Watson and also did the (unintelligible) conference but now we're focusing more on adding, starting adding these into our portfolio so people can start like, using it and start exploring with this technology. 

Benjamin Resnick:    Our goal is to make this available to IBM Global Elite clients, (unintelligible) and let's talk about the future a little bit.  In the next 10 years, we see big changes to design industry, really disruptive changes in how (unintelligible) operate. 

                              There's so many infrastructure tools being made that are making it easier for a single developer, a single designer to create product and you see that (unintelligible) becoming less and less about technological limitations because they're being overcome and more about (unintelligible) user experience. 

                              It's a very limiting time to be a designer because you can pick up the creative solution increasingly you can make it real.  So we're excited to continue working on this project and moving it forward. 

Al Martin:            So here's what I heard.  So it's not quite available yet though you can get information like on YouTube, et cetera, we'll put that in the show notes.  You're looking to integrate it with the data science experience for those listening that is our ID8E in analytics that really enables or yes, enables and empowers the data scientists. 

                              We're going to release the elite clients first and then I thought I heard you with in the dialogue saying hey, you're willing to partner with clients or data scientists that have ideas that, which they'd like to explore.  Is that what I heard right? 

Benjamin Resnick:    Yes we would certainly love those people to reach out to us.  

Al Martin:            So where can the audience go to get more information?  I mean, where can they find you guys? 

Alfredo Ruiz:       Well they can definitely reach out to, we can put our e-mails there and also we have the medium accounts where they can see the way we write and I have a Twitter account, AlfredoRuizC

                              So yes, reach out and we'd love to talk through.  I would also like to add like, we are also in the wrong right now and we're finally seeing a couple of designer works and so people want to help us with their vote and that’ll be pretty cool. 

Al Martin:            So here's what we'll do, we'll put your medium in and your Twitter accounts in the show notes so people can find you.  Look, you know, my kids are grown up now but I'd like to see the VR world get the point where everybody is off the couch versus pushing buttons.  

                              They're playing like a sports game in a. in a virtual reality room with a virtual team.  When's that going to come so we can see the kids get off the couch?  That's what I want to see.  

Woman:               Me too.

Al Martin:            So let me finish up with you guys and I always like to do a little lighting round or speed round to get to you know a little bit.  It's always interesting for me particularly in terms of doing some follow up research and so my first question, you guys have been doing a great job. 

                              Anyone can answer but my first question is what are you currently reading or what books would you recommend? 

Alfredo Ruiz:       Well I'm currently reading, it's called The Fourth Transformation.It's by Robert Scoble and he's talking about this new era of we’re living and how we're changing from you know, from smartphone to now these types of technologies and I think it's pretty interesting.  

                              Another great book that I'd recommend if you like this, technologies, Ready Player One.  It's a science fiction story and it's story about the future in how the world is going to change when we start experiencing with all these technologies. 

Al Martin:            Very cool we'll get that in our show notes as well.  Any others? 

Benjamin Resnick:    I've been reading biography by Sam Walton “Made in America”. 

Al Martin:            Yes. 

Benjamin Resnick:    It's a little bit, not related to Augmented Reality but I'm enjoying it.

Al Martin:            No that's alright.  What do you enjoy about it?  I mean, this is, like, there's no worlds here.  It doesn't have to be augmented reality.  What makes that interesting to you?  I've actually read it to you too but what are your thoughts? 

Benjamin Resnick:    I think it's really interesting to compare how he scales his business and how he thinks about his employees in particular what struck me and how they create value versus how values being created now by companies like Amazon in the digital age.

Al Martin:            Nice.  Great.  Well, two more question.  One is so what's the next big thing that you guys want to do?  It can be personal, professional, just curious what's the next big thing on your list? 

Alfredo Ruiz:       I think for me right now, well I have a one year and a half daughter so it's a really fun experience right now to, it's my first kid so it's great to experience and you know, be around her and see how she is you know, changing and it's really, really fun to, that's it.  So I'm focusing on that and well, I definitely looking forward to see the evolution of you know…

Al Martin:            It changes priorities right?  You make sure you dedicate time because they told me and you never believe it but time goes fast.  I promise you that.

Alfredo Ruiz:       Yes definitely.  Yes and you mentioned about your kids and how they you know, they're playing video games and all of that? 

Al Martin:            Yes. 

Alfredo Ruiz:       We went to a trade show, a conference, and we saw some of the new coming VR games and they are really cool.  Like it's definitely going to change the way we play games.  It's really pretty interesting.  

Al Martin:            Very good. 

Benjamin Resnick:    I'm a technological optimist but if I had kids I would not let them touch VR until they're at least a teenager. 

Al Martin:            Well that's, why is that?  I might as well ask.  You wouldn't let them touch VR because of…

Benjamin Resnick:    It's incredibly immersive and it's, I think it's just another level of like, video games at least (unintelligible) not occupying their whole world.  I think it's important for kids to learn how to like, just independently technology and be social (unintelligible). 

Al Martin:            See the difference is when I was young, we actually used to go outside and play.  I don't know that anybody recognizes that there is a door that leads to the external world.  They'd have to call me in and find me.  

                              Okay last question, you know, I did a key note yesterday and I must have had 10 new graduates come up and I probably wasn't the best one to give them advice but they wanted advice on where to really become an expert in data science.  They were saying should I get my masters?

                              I mean, stuff that you know, is personal decisions but you know, you're in the data science field, in terms of becoming an expert and really I don't know, pushing the envelope can you give to the folks that are nearly graduating or about ready to graduate? 

Alfredo Ruiz:       Well I think like I'm not an expert in data scientist, in data science but I think the new tool that the you know, are around, like a good example (unintelligible) like data science experience, I think they're fairly easy to use so they can start exploring by themselves and started learning.

                              You know, we have this thing called internet that you can basically see like, get information of everything and learn a lot from there so yes, I think that's a great starting point and yes.  

Benjamin Resnick:    My advice would be use Cortera and use online frequencies because like in this day in age to get to an interview, you just have to have the skills I think, I think diploma with a name on it matters less than it does now for a technical position.

                              And there's great resources to learn incredibly valuable skills and not say anything on the internet.  

Al Martin:            Fantastic.  I think I'm out of questions at least for now.  I mean, I could keep going on and on as you might imagine but I always like to give you guys the last word in case there's anything that I miss or something you're passionately, you know, you're passionate about that we didn't say.  This has been a great conversation.  Any last parting thoughts?

Alfredo Ruiz:       Well no, I guess.  We really thank you for this interview and we are definitely looking forward to keep showing the way we're embedding and how details can definitely be a game changer for a lot of people for the industry. 

Benjamin Resnick:    Thank you for your time, Al. 

Al Martin:            Hey pleasure.  This was great.  I'm going to follow back up with you guys separately and learn more about this stuff.  So look, and like I said earlier, your contact information will be in the show notes so folks can hook up with you.  Thank you so much.  This has been fascinating.  Talk to you next time.