Making Data Simple: Inside machine learning with Steve Moore and Jean-François Puget
How does Artificial Intelligence (AI) come into play on a day-to-day basis? In this episode of Making Data Simple, Jean-François Puget, distinguished engineer, machine learning and optimization, and Steve Moore, senior story strategist for Inside Machine Learning on Medium, join host Al Martin to talk about machine learning, AI and more. A leading expert in his field, Puget discusses how to differentiate between machine learning business problems and optimization issues and how to get more value from your data.
00.00 Read "5 tips for machine learning success outside Silicon Valley" by Jean-François Puget.
00.45 Connect with Steve Moore on LinkedIn
01.00 Listen to Making Data Simple: Machine learning for dummies with Judith Hurwitz and Daniel Kirsh.
01.10 Listen to Making Data Simple: Will machines take our jobs? With Jorge A. Castañón and Óscar Lara-Yejas.
01.00 Learn more about IBMs ML initiatives here.
01.35 Learn more about what optimization is here.
04.30 Want to know what DSX means? Find out here.
04.40 Learn more about CPLEX.
07.40 What are neural networks?
07.50 Learn more about deep learning here.
09.02 See the capabilities of neural networks.
15.00 Learn about the Kaggle Competition.
15.10 Read "What IBM looks for in a data scientist"by Seth Dobrin and Jean-François Puget.
15.20 Read "So what exactly is IBM doing differently with machine learning?" by John Thomas
25.40 Didn't watch The Terminator? Watch the trailer here.
33.00 Read "How IBM builds an effective data science team" by Seth Dobrin.
33.05 Follow Inside Machine Learning on Medium.
Ready to dig deeper? Check out our previous podcast episodes of Making Data Simple
Al Martin: Hi folks, this is Al Martin from Making Data Simple, the series, if you will. Today I have Jean-Francois Puget. How’d I do?
Jean-Francois Puget: Yes, you did great. You passed your French test.
Al Martin: All right, good, I’m going to give you the [name] JFP from now on, is that all right? So JFP is the distinguished engineer for machine learning and optimization, that’s the topic today and we’re going to go into that. I also have with me [Steve Moore], who is a senior content designer and storage strategist. Hey, Steve.
Steve Moore: Hey, Al. Hey, JFP
Al Martin: So Steve wanted to join the conversation, ask a few questions. So he’ll ask the intelligent questions, I will ask the normal, blockhead questions, if you will. So, thank you for being here. We’ve done a lot, well we’ve done at least, I think two podcasts on machine learning. We’ve done one on machine 1.15 learning for dummies, one for IBM machine learning, how to [help], if you haven’t heard those, go back, so we can’t do enough, and I notice that on your title JFP is machine learning and optimization.
So I guess my first question for you is, you know, you’re one of the IBM experts, the evangelists for decision optimization, and particularly in the optimization part, what is it, and why should we be paying attention?
Jean-Francois Puget: All right. So people hear machine learning and deep learning every day. They have a good grasp, but it’s about predicting the future, or seeing the future from [today]. Optimization is moving one step futher, for instance, retail can use machine learning to predict demand for a product sales forecasting. You can have a good grasp on what your sales are likely to be in the next week. You can use optimization if you have good focus at your space. You can use optimization to manage your inventory, to know when to replenish, and manage your inventory at the lowest. So optimization is about making business decisions that improve given business goals without changing cost of inventory.
Al Martin: So that is to say you take the output of machine learning, and then optimization is simply making that and driving business value.
Jean-Francois Puget: Exactly, exactly. Taking advantage of what you know about the future, and plan accordingly. So planning scheduling, are use cases for optimization.
Al Martin: Perfect, so that takes me to the next simple question: so what are you working on these days that aligns with that concept?
Jean-Francois Puget: We have a strange phenomenon coming now, especially in the ML Hub, they say “Oh, we want this machine learning, and here is our problem.” Nearly half of the time, their problem is an optimization problem and not machine learning. And we have to explain them, you look at the right goal, indeed, you have an interesting problem, we can help with you. But believe it or not, or rather believe it, what you need is not really machine learning, it is another analytic technology called decision optimization. So we need to find — I’m working, you know, on how can we capture this wave of business learning, thinking they need machine learning when they could use optimization instead. So that’s what I’m working on now.
Al Martin: So are you suggesting that when they come to you, they think that machine learning is the issue and you say, “No, you’re confused as to what the definition of machine learning, optimization is. You’re on the optimization side.” Can you work to describe technologies to help with that?
Jean-Francois Puget: Yeah, and the problem is, it's hard to tell people that you’re confused, you don’t know what you want. So my gut feeling is that we have to present optimization as a flavor of machine learning instead of fighting the will of people. Instead of saying “No, you don’t need machine learning, you need another thing,” I would prefer if we can change optimization, and some of our competitors things that’s what they are doing. In their machine learning use case, they need optimization. Everybody’s happy.
Al Martin: That was actually one of my questions, so you phrase it such that optimization is an extension of machine learning, so that makes the very receptive to it, say “All right, now next steps,” and then what is the next step? What’s the technologies you’re typically prescribing?
Jean-Francois Puget: So we would prescribe our product or select, which are market leading products. In the optimization arena, operations research, that’s the new wave of research going to machine learning. You don’t know if they can benefit from it so that’s our challenge today but we have the right technology, to say where can we get the source, it’s ready to be consumed by the system, but we need a way to enable to the customers, so our team, to position optimization where it should be.
Al Martin: So what’s the power of CPLEX — and by the way, those listening, DSX, the data science experience if you listen to previous podcast. So what do you see the power of that technology that differentiates itself because it is market leading. Why is it market leading?
Jean-Francois Puget: One reason is that it is not commoditized by open source for other things in analytics the best things are found in open source. For optimization, it’s not the case. For serving a problem that was basically, I’m not kidding something that was taking 20 something hours took two minutes so some number of hours versus two minutes, you see the value so that’s the reason. It’s because it gives a fast turnaround.
Al Martin: But I presume you’re still a fan of open source in general.
Jean-Francois Puget: Yes, so for machine learning and, I’m really a fan of open source, that’s where the action is, optimization, not sure why. I know why. Because CPLEX is free for academics so there is no incentive for them to develop anything in open source for this community. For machine learning, yes, I’m a fan of open source.
Al Martin: To get into more a little bit of the taxonomy, and I’m after I think previously in a podcast, so I think, it’s just so confusing, it gets confusing for me, even, I’m in the business. We talk about there being no AI, artificial intelligence or augmented intelligence, without machine learning. No machine learning without analytics, and no analytics without data. I’ve got a couple more questions on that.
First one is, what do you, what kind of AI do you think, is powering some of the decisions on optimization?
Jean-Francois Puget: Yeah, so artificial intelligence definition has evolved over the years. Today it’s a synonym with deep learning and machine learning It was not always the case. In the ’80s, people old enough like me, we have the (unintelligible) and the AI was about trying to replicate human reason. And even earlier, started as a tiring test that AI was a system that was able to trick pretending that a few men were.
So the definition of AI has evolved today, the ultimate goal of AI is to organize what’s in pictures and videos. It won’t stay that way, so a reasoning is part of AI, this is kind of sidetracked today, but we believe that optimization can be the call, the reasoning of AI, and we want the position it that way. AI includes machine learning if you confident you can reason, sure. But if you cannot reason, you’re not intelligent either, so you need learning and reasoning, and we believe optimization is one form of example that we learn that way.
Al Martin: I think it’s confusing for the listener, and I’ve actually heard this from some of the listeners, and that is the difference between AI and deep learning.
Jean-Francois Puget: So, I agree it’s confusing. So it’s very simple. Deep learning is a set of technologies called neural networks that is good for image recognition understanding speech and videos, understanding what is in videos. So this is processing what is structured data and learn from it, and so that is deep learning. Deep learning is a part of machine learning.
Now other forms of machine learning have for structured data, and then deep learning is a form of machine learning and machine learning is one of the capabilities of artificial intelligence. AI includes machine learning, it includes reasoning. You know that Deep Blue, playing chess, AI, (unintelligible) is the IS working, not just learning So AI includes reasoning, includes natural language process, includes natural language processing, and it includes machine learning.
Al Martin: So you’re suggesting deep learning doesn’t include the reasoning part?
Jean-Francois Puget: No, deep learning is just about processing, deep learning is about replicating the fact of human vision, hearing, mostly vision and hearing but its mostly, Watson has other things so deep learning is more about seeing and hearing, so these are deep learning as well. So deep learning is about replicating human ability to process sensory inputs, the eyes are a sensor, the ear is a sensor, so it’s an important piece of intelligence, but it’s not all.
Al Martin: Just as another taxonomy, neural nets being what, nodes that are interconnected that just model human brains?
Jean-Francois Puget: No, so, deep learning and neural networks, it started that way, that way in the '60s, when people are trying to replicate, simulating in computers how the brain works. So they start with neurons and connection between neurons. But it has evolved in a way that now this analogy’s no longer true. What we call neural networks, but they’re really actually — and you’ve heard of matrix multiplication — and deep learning really a bunch of matrix multiplication. It has nothing to do with how the brain operates.
People are still using analogy to get the words and headlines, but no. Because it has, some people are still working in neurosciences, trying to replicate the human brain, but it’s different from deep learning.
Al Martin: So, you referenced Deep Blue, certainly we have Watson, Arthur Samuel was one of the first out there in IBM. I said before, and he created Checkers, which was —
Jean-Francois Puget: It was the first machine learning program.
Al Martin: Exactly, exactly. But I guess the question from, to extend that further, I know IBM is doing a ton more, as it relates to neural nets, deep learning, et cetera. I don’t know if we need to do a better job explaining that at times, or you think we’re doing well in that and we should you know, at least talk to some of those technologies that are shaping the market.
Jean-Francois Puget: So we have an issue, not just a small technologies. So there is hype and then there is what people actually need. Hype is like in deep learning to recognize cats on YouTube videos.
Al Martin: I have seen a lot of cat recognition by the way.
Jean-Francois Puget: Yeah so it’s fun and everybody can relate to it, I say cats but it can be anything of interest it has a value, for instance, it certainly has some value. We host it for them and we initiate. They can learn from it. And the technology they need to learn from is not deep learning, it’s other machine learning technologies. The problem is those technology has not made headlines, so we have to educate people on, yes, tones of flow and deep learning is great, but do you really have an image recognition problem or something else. And if it’s something else, then you need something different than deep learning.
Al Martin: You know, I lead hybrid data management. Which is development and support for all the technologies in hyper data management, and we’re putting ML in everything we do. This is a question for you, not for me.
So where I’m going with this is, you know, what do we offer that’s so special? That you think differentiates itself in the market? We can tell IBM — I like to be agnostic on this podcast as much as I can, but sometimes I like to call IBM when its due — but what are we doing that maybe most out there don’t know of, that differentiates us, and when they think machine learning they think about taking on any IBM technology? Again, in my case, I’m putting them into the database, et cetera, why should they be thinking IBM, what’s the differences?
Jean-Francois Puget: Yeah, so, as I said for machine learning, open source is great, and one of the piece of learning which is to create what people notice now predictive models. But once again, the problem of our customers and you and your team is we have data, we want to learn from, how do we go from this sets of values, and creating a model is easy but its about creating the structure. What you need how do you select among the different models, how do you consume the model, how do you embed them in your applications?
How do you monitor how your model performs? Because machine learning is not a one-time thing, you don’t rent a model and then that’s it. If you have a model that learns how people behave, the behavior change over time, you need to refresh your model.
So what we deliver is what I call DevOps form, going from development of models to their operation, to the embedding in where they are needed in the application in targets, and all the production chain going from the raw data to creating models, differing them, monitoring them and adding awareness.
You don’t want a bright intern to suddenly deploy models in your production system that may impact your business You want to have some governance and flexibility of what you do. So that’s what we provide. We used open source for what it does, creating models, and we do all the management around it that can provide this.
Al Martin: How long you been doing this? Optimization —
Jean-Francois Puget: Oh, I have a Ph.D. in machine learning from the previous millennium, so.
Al Martin: You’ve seen a lot of change.
Jean-Francois Puget: Yeah, so it’s a long-time interest for me, but it was not interesting for many people at that time, so I fought and I moved to optimization. I been back to machine learning three or four years ago.
Al Martin: This is like the pinnacle. You get the Ph.D., most people don’t get it, now they’re starting to get it, not you’re at the top, everybody wants your, you provide consulting most of the time, is that what you’re doing? Are you getting your hands dirty doing some of these models?
Jean-Francois Puget: Yeah, so I have been a manager and developer a long time in optimization and in machine learning, I’m — no longer being a managing developer — I’m providing inputs on what people need to do, so from the tooling what are their needs and I’m also building models for our customers, working with the data science team. I also do more, I am not getting enough machine learning at work so I also do some on a site called Kegger, a machine learning completion. I encourage all wanna be data scientists to check it out.
Al Martin: You must do a lot, then. So, you know, everybody’s talking about autonomous cars for the broad detection, a lot of insurance companies, you know, they’re talking about fraud detection, medical diagnostics, one of my keynotes, one of the ones I like to talk to is, like, skin cancer, detecting cancer and how much machine learning can help with that. What trends are you seeing? I mean that we’re not seeing out there in addition to those? I mean, all those are fascinating to me, but I can see more coming around the horizon.
Jean-Francois Puget: Yeah, so you mentioned healthcare, you mentioned fraud detection. These are great use cases of machine learning, and to me they’re more interesting than cats on YouTube, but that’s me. I believe there is a trend that we don’t speak too much about in general, it’s machine learning to secure, because fraud detection and intrusion are similar, if you take them by detecting anomalies in something. Same for cancer. So all those use case having someone, they are anomaly detection, and machine learning is a great bit point, because you use machine learning to learn what is normal, what is regular, and then you detect the deviation from that.
So machine learning applies to security, intrusion detection, is I think a great opportunity, our security division which is in machine learning, so that’s not something people speak much about. The other thing that people don’t mention but everybody is knows about is the recognition system.
If you go on an e-commerce site, you start putting stuff in your basket then the site propose you saying “Oh, other consumers. These are recommendations," and they build on past sales automatically so you have a learning system that builds a model that predicts for every visitor, what are the products, he or she is likely to buy. And they recommend, and that’s the most common use case for machine learning. Again, it does not make any headline, but that really a serious business recommendation system.
And I believe this idea of recommendation can be expanded to not just e-commerce, but we are discussing with one of our product teams, can we use machine learning to make the use of the tools easier when the mainframe is proposing actions to the user, trying to understand what the user wants to do, and then propose next step instead of having the user discover into read a long language instruction manual what to do next. So use machine learning to recommend actions and make our products easier to use.
Al Martin: Very good. Steve, you got any comments, any questions, you’d like to chime in? I’ve got more, I’ll just keep going, can’t shut up, so what do you think?
Steve Moore: I’m curious to move back into optimization for a second. I wondered, JFP, if you could talk about one or two examples of how that’s really happening in the field and maybe talk in particular about the kind of data that we’re finding to feed in.
Jean-Francois Puget: I will give you a use case which is company, so when you buy or sell on the market your sale looks like it’s so called executed immediately but actually it’s not. You’re just, your sale is recorded and your bank points and see if you’re selling your bank earning your equity. We talked with the bank of your buyer, who has money, and they make sure that your money swaps with counterparts that’s what its called curing. This is basic optimization, you want to match buyers and sellers.
So you know people are saving, are shopping, so they are selling stock people are buying with borrowing money, so at the end you must make sure that people who buy have the money and people who sell have the equity. You try and match as many sales as possible, that’s an optimization problem. And improving on this process is extremely large return, so we did the one project for the Mexican central bank.
They switched to our technology for clearing the after-market space, and by doing it, they were matching more sales, they were reducing the need for short term borrowing money. They claim that they saved that is $250 million a year. We did a similar thing for European central bank and the savings are billions a year. But the best, when there is a private clearing house company, I cannot name it unfortunately, but they told us that they are paying the cost of the optimization project, every minute. The return on investment is so large that in one minute they make what and they paid us 200 case, so the return is 200 case every minute. I don’t know of any of them technology that creates this kind of return.
So it’s a hidden gem I call it IT's best-kept secret, because it’s really a very good technology, but it’s underused probably because it’s too complex to use has been making it easier to use and maybe machine learning is one way.
Al Martin: What do you think so going forward, you’ve done this, how many years was it? I don’t want to date you here.
Jean-Francois Puget: No, so really working on the machine learning of things lets say two years. Optimiztion is way longer.
Al Martin: So you’ve seen a lot. Where do you think, just in the industry, where do you think ML, AI is doing well, where do you think, you got to be like, you got to be, like any technologist, frustrated in some areas.
Jean-Francois Puget: So I think the main for people working as data scientists, you know, that there is a lack of data scientists people are fighting for. And one reason is the technology as good as open source it is very time consuming is to do a lot of trial and error, it’s consuming a lot of time, so one of the priorities we have this year is to work on automation and so that people can perform the same quality of machine learning model by spending much less time.
So we’re working on the automating the work of data scientists and that’s a trend in the market, that is maybe in 10 years we won’t need data centers anymore, I doubt it. Be more productive. They would concentrate on where they add value and less on that that task that we automate for them.
Al Martin: You say in 10 years, what changes in your mind or what could change in ten years to make that job, not to scare any data scientists out there, but make their job irrelevant at some point, every job goes irrelevant at some point, perhaps, but what do you think’s going to happen in 10 years?
Jean-Francois Puget: I think we will have autonomous learning system, so it we speak about autonomous cars they don’t run by themselves. Its centralized for now. but I believe one job of machine learning is to be able to automate work of the data scientist in some cases so we no longer need them and we can have an application that learns. For example, I visit a website and go on the same pages and present it to me. It looks like so simple, and you don’t need rocket machine learning for it, but you need to have a system that learns automatically for each customer, and adapt to each customer. So I think that that will be a natural variation of machine learning to have a system that adapt to their user, automatically, and this can go from car to your coffee machine to whatever you use network on.
Al Martin: So if it’s going to take 10 years for that —
Jean-Francois Puget: No, maybe less.
Al Martin: Maybe less, all right, maybe less. But still, I was going to, there’s this dream and maybe fear of artificial general intelligence. And that’s when, that’s called the holy grail of artificial intelligence, where either take over the world or…
Jean-Francois Puget: Yeah, Skynet and Terminator, And you know, I’m old enough to have seen this in the past. And you know, when people need funding, speak about Terminator. They need funding so they ask to make that not a threat anymore. So that’s what the game is, but again, if we look at how AI has evolved, in the '80s, researchers were trying to capture human reasoning in expert systems. Today, researchers are trying to capture things that any kid do without thinking. You know, recognizing what objects you’re looking at, which those kids did learn very quickly.
Recognizing speech, we all do it, almost all, without thinking. And that’s what we call AI. So the bar went from highly demanding intellectual task, to what kids do without thinking, I don’t I don’t think we’ve made much progress at all in general AI. We’ve made progress, very specific domain, we’ve made great progress.
There’s no comparison between autonomous cars we’re driving, we’re making progress. But does it require a lot of intelligence to be able to drive? If it was the case, we would have less cars on the road, it would be selective, anybody can learn how to drive, so it’s not, but progress is steady in that space, that’s for sure.
So augmented intelligence in specific areas yet, and we see more of it, the general AI, as long as there’s no real investment, reasoning and planning and going beyond the next set of actions, we won’t see much progress. So I don’t think we’re making progress towards general AI. I don’t think the terminator is coming at all I don’t think I will see it while I am alive.
Al Martin: Yeah, the good news is, my mother-in-law listening out there, anyone can eventually learn how to drive. If she does listen, just kidding, just kidding. Hey, Steve, anything else you got? I want to ask JFP a couple more questions about his roles specifically thought I might give you room for a question or two.
Steve Moore: Sure, one of the things you said JFP just a minute ago made me think about the degree to which IBM can start to use machine learning even in our own processes and the dev processes and understanding what customers want by getting feedback about how they’re using the product. You see that as something that’s starting to infuse into our own.
Jean-Francois Puget: Yes, so we have an initiative we call ML Everywhere all of analytics is really looking at, how can we can eat our own food and how we can use machine learning and optimization and more generally, or even technology to make our tools easier to use how to automate repeats so that they can focus on (unintelligible). So there is definitely, we already are doing it but want to do it at scale and really focus it.
Steve Moore: …alternate road, but so much of what we do day to day is about figuring out what context we’re in at any given time, whether we’re driving or reading or walking the dog or or talking to a friend.Is there some idea in the machine learning community about how we teach things to establish context or know what context they’re in, what the draw on from their school of intelligence.
Jean-Francois Puget: Yeah, no, frankly the success of deep learning has more researchers into one specific area of AI. Today, there is not really progress in what you mentioned. I speak about reasoning they tried all in that area except some outliers. But I believe there is an outlier and there is not so much left to be done and speech recognition, yes we can improve a bit more the product. This is becoming engineering and more research, so I would not be surprised if the AI community, the AI research community moves back to more general AI.
Al Martin: You know, oh, go ahead. Go ahead, Steve.
Steve Moore: I was just going to say thanks, that’s interesting to hear.
Al Martin: What’s the most exciting thing about your job or what did you like about?
Jean-Francois Puget: I could say while I’m at IBM, it’s no surprise to people in this kind of job. I get offers from lots of companies, including Silicon Valley, but I prefer to help people fight cancer or detect security breaches. I think the most exciting is the kind of problems we have access to at IBM. We have a unique range in the industry. We have a variety of issues we can work on and how to help the world than what the other companies may have.
Al Martin: Yeah, that’s a great answer. To that point. I was, I don’t know what the specifics are so I won’t quote them here unless you have them, but I know that the oncology team has, I don’t know how many algorithms for cancer right now, they don’t get a lot of press because it’s not like cats on YouTube used to, right, but I mean solving real problems that gets me excited.
How do you keep on top of all this stuff?
Jean-Francois Puget: My area is Kaggle, because there you compete with the best. So you can tell if something works or not right there. So to me that’s what I do. I don’t think machine learning is something you learn by reading, you need to read to learn, maybe go on force but you need to practice, because it’s a very quick developing field.If you think about it, before 2013, nobody was thinking about this deep learning. Nobody was speaking about deep learning. It is developing very fast. So for me either you go to conferences, and there is a conference every week, I don’t know when you work, or you practice and you compare with others and learn from others. Please practice. It’s not so difficult to start.
Al Martin: I was going to ask you that exact question, can I go out there, and anybody that doesn’t have the skill, can I start there? Or is that over their head.
Jean-Francois Puget: Yeah, no, no, so researchers in machine learning, they have Ph.D.s, you can really start, you start from an exact word and you adapt to your problem, and you can in weeks you can become a good data scientist. I have seen it. I am not saying everybody can do it, but certainly some people can.
Al Martin: So you must be taking all the stuff that you gather, though, and I know that like, keep you honest here, Steve, because I know you’re interested in this as well, the inside machine learning hub you’re one of the prime authors out there, correct? I mean so you’re writing a blog, you know, that correlate to what that we discussed here.
Jean-Francois Puget: Yeah, I try to learn from this, because if you look at what’s available you have research papers, you can read all state of the art. Then you have very high level and simplified blogs and clickbait type of blogs. paper saying, “Oh, we can have your hack your blah blah blah.” But I think something in between that provides value to practitioner and researchers. I try to blog useful stuff for non specialists.
Al Martin: Fantastic and Steve, I don’t know if you have anything to add on where to go to gather that information.
Steve Moore: Yeah, so Information Learning is a publication on medium.com, and JFP we’ve got people from inside analytics and outside the community, so it’s a great resource as JFP said, we’re really trying to appeal to people who are fairly technical already, on the one hand, to give them real information and then, also the people that might interact with machine learning, JFP for example, wrote a great post that talks about what are the skills of a great data scientist, what are the skills for a data science team.
So that’s useful to people who are trying to put that set of skills together to really understand what separates somebody who’s a newbie from somebody who really knows what they’re doing. So that’s hope, really helpful to people.
Al Martin: And by the way that Seth you refer to is Seth Dobrin, who’s our chief data officer. Very good. What do you do for fun? Is it all machine learning, all the time? There’s got to be something else you do.
Jean-Francois Puget: I live in France, that’s fun.
Al Martin: He drinks wine all the time.
Jean-Francois Puget: All the time, it’s funny each time people say, and tell me when I am on the phone, “enjoy your drink”, so I don’t know how I look, but I don’t drink wine all the time. I drink water too.
Al Martin: Just not very often, right?
Jean-Francois Puget: No, but I live in a very nice, I live in south of France near the sea so it’s a bit like California, same kind weather so I hike I go swimming, so when I am not doing machine learning.
Al Martin: So for those listening, we’re sitting, we’re both in Silicon Valley right now, beautiful day, that’s what he’s referring to.
Jean-Francois Puget: Yeah California is a place where I could live, but I like where I am.
Al Martin: But you do know that Napa wines are better than French wines.
Jean-Francois Puget: I agree.
Al Martin: You agree?
Jean-Francois Puget: I’m not really strict. We have great wines, but fortunately not red wines California wines are very interesting. They are like bottle wine. They are quite interesting.
Al Martin: Well, look, I got a lot of information out of today’s dialogue. Thanks, Steve, for Joining me and JFP for joining me.
Jean-Francois Puget: My pleasure, thank you, and I hope some of you will get some useful information out of this.
Al Martin: Well external, you know LinkedIn, you got Twitter.
Jean-Francois Puget: I’m actually on Twitter, I’m on LinkedIn.
Al Martin: All right, we’ll get that out there, we’ll put it in the notes. So thank you for joining, thanks to (Steve). We’ll sign off.
Jean-Francois Puget: Thank you.