Data Decoded E5: Evolution of Technology
Floppy disks, punch cards and BASIC — we’re taking a trip down tech memory lane. How has technology, specifically in terms of data management, evolved in the past few decades? On the latest episode of Data Decoded, Andy Leonard, founder and chief data engineer at Enterprise Data & Analytics, discusses what patterns he has seen in the past, what we can predict about future trends in data management and how data professionals can keep up with technology constantly in flux.
00.10 Need help tackling Big Data? Get guidance from a data expert.
01.27 Check out the 2018 data predictions we made in the beginning of the year.
02.26 Remember the Motorola 6800?
03.03 Gain confidence in your data with data integration.
03.40 Read 90 Days to Consulting, written by our host William McKnight.
05.44 Remember Burroughs B80?
06.06 There is no artificial intelligence without information architecture.
08.59 Take a trip down memory lane with floppy disks.
09.36 Real-time business value can only come from real-time data.
10.28 Dumping data into a Hadoop data platform alone won’t accelerate your analytics efforts.
10.57 A New Approach to Cloud – IBM Cloud Private for Data.
12.44 Your data infrastructure needs to have a trusted data foundation.
14.19 Never settle, you & your data deserve better. Schedule a consultation on how best to govern your data.
16.20 To keep up with the changing industry, you need to ask your data these 5 fundamental questions.
19.16 Offload your ETL workloads to a data lake or Hadoop.
21.43 Learn more about IBM’s journey to GDPR readiness.
22.12 Manage your data at a high level.
23.08 Listen to Andy’s podcast, "Data Driven."
23.22 Learn more about the team behind the podcast, Unified Governance & Integration.
William: Welcome to Data Decoded, an IBM podcast series dedicated to demystifying the world of data – from data lakes to master data management to big data and everything in between. This is the podcast for the data professional and all those who understand they are in the business of data. I’m your host, William McKnight, President of McKnight Consulting Group.
Today we’re joined by Andy Leonard, the Founder and Chief Data Engineer at Enterprise Data & Analytics, data integration trainer, consultant, developer, and data warehouse developer. In short, Andy wears many hats and we’re quite lucky to have him on our show today and I consider Andy a friend. Welcome Andy!
Andy: Wow, William, thank you so much for that warm introduction. I consider you a friend as well. It’s a big honor for me to be on your podcast and thank you so much for having me.
William: Well, let me tell everybody what we're here today for because it's a little different. Usually we’re talking about technology issues of the day. We’re going to take a look in the past though, with the evolution of technology and how we got to where we are today and hopefully it will be a little nostalgic for some of us that have been around a while. But as importantly, I want look to the future with you, Andy, and see what the past foretells for the future.
So I have chosen you to do this journey with me here today, so tell everybody about yourself a little bit more and how you got into technology.
Andy: Well, I think you made a wise choice because I'm really old.
William: You're just a couple years older than me so please don't say that.
Andy: You know, I say this, you’ll get this, when we were kids – 55 was when people were in wheelchairs, they were walking with canes. I’ll be 55 next month and –
William: I take it back, you’re a little bit older.
Andy: So yeah, I thought so but you know I ran a half marathon just about five years ago. I know you're into a lot of fitness and stuff like that. So things have changed. I mean there's a lot of changes. Technology is just one area where they changed.
I started at the tender age of eleven. I learned Motorola 6800 machine code from a neighbor who had retired from the Air Force. Later that year, I learned BASIC.
You know, it was a hobby. Software was a hobby until maybe the late 80’s early 90’s, where it started intersecting with work more and more. Since then, I've jumped to fence the data in the early 2000s. I was a developer prior to that time and I do a lot of data engineering / data integration work. I started like everyone in this field, focusing more on the cloud. There's just so much out there and we see the same-old new things kind of popping up, right? The patterns just keep repeating. So I'm looking forward to chatting with you more about how things changed over the years, what I've seen. I'm sure you've seen some of the same stuff, some different stuff and I'll just throw this out to you, William, while we're here.
Maybe people don't notice, I'm a huge fan of William McKnight. I read your 90 Days to Consulting success book. I reached out to you after reading that book. I learned a lot from that book. I still have a lot to learn so I’m looking forward to picking up some more tips as we talk.
William: Well very good. Well in the past 30 years, which is kind of a time period I think we can both relate to, if we kind of peg where technology was back then. Back then, I was joining IBM and you were doing what you just said you were doing. But from that point until now, what are the major shifts in how organizations approach technology?
Andy: Well gosh, for enterprises there was a lot of mainframe stuff out there than there still is. The machines themselves, RAM, ran a lot slower than they do now. I want to say that was the beginning of the client server era when you saw a lot of early steps in distributed computing. I refer to those days William, as to when we used to carve our own chips out of wood.
Back when the years began with a one. You know this, but it was a lot different. There were a lot of green screens. There was the beginnings to graphical user interfaces. Just a different world. What's interesting though, is how that's evolved. Right, how all of that has evolved and what has led to now. And as I said earlier, I see a lot of the same patterns. Like take for instance, mobile. Today it appears a lot like a big evolution of client server.
William: It sure does. There’s been so many changes and languages in use and the time it takes to do things. I mean, my first computer class was in high school and then my junior year of high school, we got this big old computer in the back. I think it was called a Burroughs B80 and we did punch cards. We did COBOL with punch cards so every statement had to be punched out and we dropped our deck on that machine and hopefully got some small thing out at the end. Boy, has it changed.
Andy: It has.
William: We’ll get there, but if we look to the future and artificial intelligence, the power that’s embedded in those applications and those algorithms are off the charts compared to back then. Let’s break it down a little bit by decades.
There was the 80s. There was the 90s. How was data managed back then? Maybe for our younger listeners who weren’t around then or not managing data then. How did we do it back in those decades?
Andy: Well you know, there was a lot of data stored on disks, hard drivers. Even today, that’s the way that it is although it’s not the same physical architecture anymore. A lot of the data being stored now is on SSDs or in the Cloud. It was an awful lot of data going to tape, back in those days for storage.
William: That’s right.
Andy: You know, the relational database management system as a concept was still relatively new in the 1980s. I remember Cod’s work was done and what was that, late 70s?
Andy: Cod wrote about RDBMS. There were all a lot of database engines out there that were not practicing normal forms and we will talk a little bit later about NO SQL and some of the databases that have been merged post-web or at least post-web 2.0.
It was a whole different world, of course things ran slower. You had to be a little more cognizant about engineering for memory – both physical storage, persistent memory, and also RAM because you remember those days – I mean we're talking megabytes of Ram and that was huge. Now, of course, terabytes is huge. So that was the 80s.
The 90s of course, things got faster. We saw Moore’s Law kick in and a lot of speed – both CPU and storage technology. And I got hooked to the Internet. In 1993 or 1994, I think it was when I first connected through CompuServ. I remember using Netscape as a browser and my search engine of choice was Alta Vista so back in the old days.
It’s hard to describe to my kids. I have teenagers and when I talk to them now about how different things were – it’s difficult. My son was learning a word processing software a few years ago and I was showing him the icons and talking about what they meant. When I showed him the save icon, it had an icon of a floppy. And he said, “What’s that?”
It didn’t dawn on me, William, that he had never seen a floppy disk. So I went to the attic, still had some and pulled out a three and a half inch floppy and said, “This is what that’s a picture of.” They just don’t know. For you and I, floppies were new for us. They were 8 inch and then were 5 and a quarter.
William: So there was definitely different medium, that’s what I'm hearing. Tape was a real thing back then. I mean we considered tape equal to you know, disks, back then. Are we going to put this on tape? We don’t do that anymore, of course, and as you mentioned, relational databases emerged.
Here's a little memory for everybody. Remember Oracle? Arguably Oracle or DB2, I don't want to get into that detail, but it was the first relational database. They were sort of back and forth there in those early days and then of course we got into the whole benchmark wars and things like that. Informix was a big player in that. And yeah, it was Informix and DB2 and Oracle.
This is all 80s, early 90s. Certainly we have a different set of players right now and by the way, going back thirty years, we didn't have the Internet, okay? So for everybody to remember, we couldn’t look things up so quickly.
And we certainly didn't have…interestingly, I'd say we didn't have the post-relational movement that we have today. You know you mentioned NO SQL, Hadoop, Spark. All these things that we have today we didn't have that of course, but interestingly if I kind of squint my eyes and look at some of that stuff today – it looks like kind of some of those pre-relational things that we had way back when, when we're looking at file systems to store data on. So, I like to say the more things change, the more they stay the same, right?
William: So now the Cloud. We didn't have the Cloud. Everybody had their own data center. There was no thought of sharing. That was just not in the picture, but now we have the Cloud. Hasn’t the cloud been envisioned for many years before it’s become commonplace like it is today?
Andy: Oh I think so and I think we saw foreshadowing of that in the early client server development that…You mentioned COBOL. Even though the terminals were not as smart as they are now, that whole idea of someone sitting at a terminal and talking to a central computer, that’s essentially the network I see, maybe the great, great grandchild but that’s the same market.
William, you know this, but there’s still bits flying back and forth. And what we saw were shifts. Some increases in technology, some really cool innovation and technology. We saw some shifts and things like, where is the line drawn?
We really saw this come to the terminal first. I think that's what happened maybe in the 90s, is that we saw the terminals get smarter and we’re seeing it again with the phones getting smarter and getting more powerful. And then we started drawing the line just about anywhere we wanted. So you know we then started injecting tiers in between client and server where more activity took place. So we're seeing that again with the Cloud. As the Cloud becomes more powerful…and really this is often overlooked as people trust the Cloud more.
Then it becomes more practical. We’ve got a whole thing in between – the infrastructure, the ubiquity of the infrastructure that’s being built. So probably where you live, William, you’re probably getting like a gigabyte or something like that. I’ve got 24, 25 mega DSL out here and I’m tickled to have it. I had 10 meg for the longest time.
You think about the things companies are going about, you know putting the satellites up, lower orbit satellites that are going to be able to talk directly to our phones. That's going to be a thing you know in the next five to ten years and that's going to really read out. If you ask me, especially as a data engineer, that's where I focus. That's the big holdup right now, is just having ubiquitous access at performing speeds.
William: That’s a great point and you know, everything that we see that is commonplace today. It's interesting they all took a similar trajectory of you know, I call it the seven stages of grief, that they all went through to get into mainstream. And cloud is certainly an example of this where people denied it, they got angry at it, and blah blah blah, they eventually accepted it, and now it’s the de-facto standard at least in many places.
Can you underscore what I'm about to say to our younger listeners? I'm going to tell them that what's hot will change and it's important to stay vigilant and stay on top of what's going on in your space. Never settle and feel like, you know, what you have learned is all you'll ever need to learn in the technology space. What do you say to that?
Andy: Well I noticed, you did that with mostly a straight face. You know at the end, when you were saying you know what you've learned is all you'll need to learn, there's a level where that applies and I will say that I think it applies at the architectural level.
Where yes, having decades of experience counts because you've seen these things before, maybe even you solve these problems before architecturally and that definitely applies. These days, I spend somewhere between 10 and 25% of my time on education. So I’ve gone out and achieved some certifications recently. In fact, I recently finished a program in April and it was directly related to data engineering in the cloud, there was a lot of open source.
The lines between companies and technologies are blurring like never before. I own a consulting company like you do William. I hire and sub-contract all the time. One of the questions I ask folks is you know, what have you done recently? What class have you taken? What online program have you done?
Tell me about the last blog post you read and what I'm looking for is for them to say something like, “Well you know just yesterday or just two months ago, I knocked this thing out.” It’s that important. It’s almost a marker of do you think you know it all and even if you know it all today, it’s going to change tomorrow. I’m not making that up. I mean there’s new stuff being released by continuous integrators all across the board, especially across the cloud.
William: Absolutely and I have a similar question. I ask “How do you keep up with the changing industry?” I do want to know what they’ve recently done, what’s their methodology for keeping up because you’ve got to have one these days. I want hear how they keep up, I want to hear they listen to my podcast, ha ha ha. I want to hear the books they’ve read, how they take notes, and how they take advantage of different learning opportunities because they’re out there for you. You can’t do it all, but you have to track things to you that makes sense in your concept.
Andy: We’re on the same page there.
William: And Andy, I mean you are on here in part because you are a great example of this very thing, right here of changing with the times, keeping up-to-date, keep investing in yourself. Your passion for this comes through in everything you do and that’s what I like to see also in other people.
What are the important shifts happening today for a technology professional? Let’s bring it into the current context.
Andy: Oh sure, so you know – cloud, cloud, cloud. Okay, that's especially in the data realm. And it's still early. We're recording this in late June 2018, so people listening in the future can't get this. What's going on today is huge compared to what was going on 6 months ago. This is nothing, nothing compared to what’s going to happen in the next 6 months or 18 months. It’s just going to continue to accelerate because – I don’t even know William, what the latest thing is. But I’ve seen zetabytes, exabytes of data. People talking in those terms and they’re talking about it in the terms of the amount of data being created at some of these popular social media sites in the world.
There’s something like 90% of all the data that’s ever been created in human history has been created in something like the last 12 months now. A few years ago, it was 18 months. So it keeps accelerating, right? So data is not going away and it’s going to move to the cloud increasingly. It’s already there. It’s already native to it. It’s going to keep moving in that direction and mobile devices – I’ve already mentioned them a couple of time already – are getting more and more powerful.
People just want to know they're going to be able to, want to look at this device, keep up with what's going on with their friends. They're going to increasingly – people are looking at the device to keep up with what's happening at work.
William: There you go.
Andy: What’s really nice about that is that let’s say we’re doing a big load, a big ETL project and we’re running that first load. Actually, I can share a true story. This happened a few years ago, I was doing a big data warehouse project and was getting ready to go on a mission trip and I left the country. I went to Central America to Honduras and I wasn’t planning on taking my laptop with me and of course, two days before…I’ve been telling people for months: “I’m going to be gone. If there’s anything we want to do, let’s do it now.”
William: Good luck with that!
Andy: Exactly, right? You know what happens. Two days before and you know this take like four, five days to run this big load. So I have to take my laptop to Honduras. I got to Honduras, William, and I had faster Internet connectivity there than I did here. I was able to remote in and monitor the load and this is kind of the story of where things are going. It’s moved from my laptop to my phone. I can work on my phone now and this is what’s coming. I don’t know what’s next after this. I don’t know if we wear a skull cap or something that reads our minds.
But it’s definitely distributing and definitely getting into that spot where you know less and less, we need to be coupled to a building, a physical location, a laptop, a cube. It’s getting more and more to where we can participate in what we call the gig economy – where we’re out there doing, king of like what we’ve been doing for a long time as consultants.
So yes, I’d say that’s the biggest thing. I don’t know that younger technology professionals struggle with this as much as I do now because of my age because my kids have cellphones and they’re use to this. As someone described recently on a podcast, they’re digital natives. I love the terminology because it really puts a line in the sand between folks like me who started typing HEX. You started typing with a bunch of punch cards. We’re not digital natives, but they are.
Yeah, just keep in mind that data to the people that are younger especially, data is exploding. It's not going anywhere. It's becoming more and more important, especially with the legislations that are coming out. We just had GDPR going into effect not less than a month ago. We don't know how that's going to affect our data and our jobs so keep an eye out on all of it.
William: Yeah this is going to be a lot more coming in there and there's a lot more than GDPR.
So we're going to have to not only have a whole lot more information, like you mentioned zetabytes and exabytes, but we’re going to have to manage it at a higher level, a higher standard than ever before as we go forward so data is job security. I love this field. I encourage people to get into it that have at least some passion for it, but the more passion and the more ways you can innovate, the better for you.
I want this podcast, I want all my messages to go out to leaders and people who want to lead this field. This field is still in desperate need of that so I think you’ve really added to that today so I want to thank you very much, Andy Leonard, for your time today. And let me also mention to the listeners that if you listen to two data podcasts…Now, the first data podcast, you know what that is, but Andy has his podcast called Data Driven and it’s a great one as well so I encourage you to check that out. So once again, thank you Andy for joining us today.
Andy: Thank you so much, William, for having me.
William: Alright well, thank you listeners as well for tuning into another episode of Data Decoded. You can learn more about IBM Unified Governance & Integration at ibm.co/UGI. You can find this podcast and more at ibmbigdatahub.com. See you later.