#108 – Artificial Intelligence as a System: Scaling AI

[00:00:00]  Tom Garrison: Welcome to the Cyber Security Inside podcast.  I’m your host Tom Garrison.  Recently Camille Morhardt and I sat down with the VP of AI and Cloud Execution at Intel.  It’s a great episode if you’re interested in how a company like Intel uses a system view for our AI strategy. Kavitha Prasad explains that our AI strategy should include items like memory management and data movement challenges, costs, what customers want and edge to cloud connections.  So let’s jump right into it. 

Kavitha, so you own AI strategy for a massive company like Intel? Where do you begin?

[00:01:17] Kavitha Prasad: That’s a good question, Tom. It is a huge charter. But if you think about AI, it’s a very pervasive workload, right? Wherever there is compute, you can have AI around it. So for a company like Intel where we are building clients to the high-end desktops, to the edge devices, all the way to the cloud, it’s becoming pervasive in everybody’s life–all the way from phone in your hand to what happens in the cloud. And Intel has the assets, the software ecosystem, everything to go make it happen. 

So it’s just a question of making sure that we are cohesively aligning the direction of AI for Intel and making sure we are executing towards cohesive direction that we have set for ourselves. There’s nothing stopping us from getting it done. And that’s what is so exciting. And it’s a huge opportunity for us at Intel to make sure that we are able to succeed in AI.

[00:02:10] Camille Morhardt:  Do you look at it as kind of a systems approach? I mean, I think you’re sort of alluding to this with Intel as having this kind of rather unique ability to approach sort of all aspects. And I’m just wondering what difference that makes, you know, the ability to look at something or consider something holistically, it seems like systems thinking kind of came up a while ago, a decade or more ago. And now it seems like it’s coming back around with full force.

[00:02:40] Kavitha Prasad: AI do consider it as a system. Because if you look at AI, right, it’s a heterogeneous workload, if you think about it.  Sometimes AI is misunderstood for deep learning, or deep learning of the inference and training. But it’s not more than that. If you really look at a real world application, you could have multiple sensors or multiple data sources; you take all that data, you do pre-processing, or you do whatever data mining data wrangling whatever needs to be done. Sometimes you do sensor fusion, which could be early or late fusion, whatever you need to do; then you do a compute on it, which could be deep learning training, or deep learning deployment. And then you have statistical ways of deploying that artificial intelligence. So there are multiple ways of doing it. 

And then you do post-compute on it. Like, you could take actions, you could do actuation, you could put a brake in your car, for example, to dropping a box off to somebody using a robot or, you know, completing a sentence in certain cases. So there’s a lot of heterogeneity to this workload; it is not just about deep learning. So in that context, if you look at it, if you look at an application, it is a system level application, there are multiple aspects to making this work. And again, at a system level, if you look at it to make this work there are multiple constraints like is it a latency requirement that is need that needs to be met? Is it a thermal constraint? Is it a power constraint? So based on all of that, how you develop the solution, and how you deploy the solution changes a lot. 

But in the end, if you look at it from a hardware system perspective, or ecosystem perspective, there are two main things that happen.  Any workload, you start off from the software, Python coding all the way to the bottom of the stack, there are only two things that matter: memory management and data movement. And this has been a problem since ages; it’s not a new problem for AI. Since compute started, these were the two bottlenecks, and they continue to be the bottlenecks even now. 

So we need to solve the memory and the data movement at a system level to make sure that we get meaningful performance out of it. And even if after you solve this, the customer KPIs are very, very different from latency, throughput power, performance per watt and all of that stuff. Customers are more interested, for example, say, how many faces did I detect in a given frame in a second? And how many pictures did I detect? Or how many units are functional in the field, as opposed to being charged in the charging station–that brings in the power constraint and the TCO costs? So the KPIs from a customer perspective are very, very different. So all this needs to be accounted when you’re defining your AI system and AI deployment model. 

Now, the other way of looking at the system is also where you have the data that is getting generated–the sensors–and you can do artificial intelligence right on the phones or the sensors or you could do it at the edge or you could do in the cloud? How do you make sure that you are creating meaningful and or compute across the entire pipeline? Because that defines your cost, too, in the end.  If you think about the petabytes worth of data that is getting generated, I have to transfer all of it to the cloud. It’s a huge cost. And what happens in areas where you don’t have internet connectivity or there is a shortage of bandwidth? What happens? So defining that system from edge-to-cloud continuum also is a way of looking at how do you deploy these AI models, effectively. 

And then to top it all, you can think of the workflows like security’ how does it play end to end to making sure that I’m ensuring the privacy if the customers the privacy of customers is preserved that security end to end, and there is safety involved in all of it, these are again the global flows that need to be looked at. 

So AI is a system either you slice it down vertically from an application all the way to silicon, or whether you look at deployment from all the way to the from the client to the cloud, to the to the edge to network into cloud, it has a system level problem. And it needs to work cohesively for us to make sure that AI is deployed effectively in the market.

[00:06:38] Camille Morhardt:  So Kavitha, people are looking at AI across so many different facets; you’re looking at, vertically, you’re looking at it horizontally, you’re looking at different workloads. There’s just so many different approaches from software to hardware, from IoT and embedded style devices to cloud devices; you’re looking at distributed models. It’s so huge. 

Are there kinds of things that people tend to overlook, or that people tend to miss when they’re packaging sort of an AI strategy together? Are there things that people miss that you think that they should be aware of? 

[00:07:19] Kavitha Prasad:  People do know this.  It’s not the question that we don’t know. But the key thing when you’re looking at deploying AI models, right, ecosystem plays a huge role in all of it. If you look at the number of accelerators, for example, that are getting developed in the industry, right, you have all the startups that are developing it, everybody seems to be developing accelerators. But one thing we tend to forget is the developer persona of this AI workload. These data’s need the data scientists are subject matter experts, they’re coding at a very high level language. So even if you develop these accelerators, if it is not easy to use, if it is not easy to get the performance out of these accelerators, nobody’s going to sit and write C, C++ coding to target your accelerator.  

The development to deployment time is very key. And that is where it’s important that ecosystem is what makes or breaks the AI accelerators or the hardware that comes into the market. It is a software problem. First, I need to make sure I’m able to get the performance right out of the box. I need to cater to the developer personas. It’s not about the raw compute that I pack into the system. It’s what performance am I going to get right out of the box? That is the key.  It’s not about the raw compute or the peak power or the average power that you pack into your hardware system. What is it that I get out of my software tool chain is the key and that is what is going to make it successful. So that is something that is going to make or break the play for AI in the long run.

[00:08:51] Tom Garrison:  As somebody who lives in this world of AI, I’m just so intrigued with this workload, because everywhere you turn, you know, people are talking about AI this and AI that.  And some of it, I think is just total hype. Some of it is you know, people trying to ride the coattails of marketing of a new hot workload. But sometimes there really is an element of truth and what they’re saying. 

And so for somebody who’s in the know, where do you think the future of AI is going? Just generally, like, what will we be able to do in 3, 5, 7 years that today is not even possible, but in the future, it’ll just be commonplace because of AI.

[00:09:33] Kavitha Prasad:  Today, we do a speech separate, we do tech separate.  But if you think about multiple input data types–images, video, speech; how do I put that together and create something meaningful out of it?  Because, for example, cat is a cat is a cat is if you have a picture or if you say a cat, or if you say a cat in Chinese or whatever language it’s a cat, how do I bring that multimodal knowledge to make something meaningful out of it is something that is going to emerge.  In a similar context, composite AI, right? It’s another way of looking at it. I have all these different sets, but I’m having all the different analytical techniques to make sure some meaningful results out of it. It could be your classical machine learning or your deep learning, or it could be your statistical learning that have been how do you build a composite of it, to make sure that the AI works meaningfully? 

And then there is another thing that’s going on is today we have artificial intelligence then there is artificial general intelligence and artificial super intelligence, if you were to put it. But artificial general intelligence where you have the intellectual capability of a human being right? Where it could be fine motor skills or problem solving recently, I think I read an article where AI models that can solve these Math Olympiad problems, which are thinking like humans to solve them, and they were 54% accurate or something like that. But Artificial General Intelligence is going to become popular.  For example, the Turing models or all of that stuff is going to become popular. 

Then on top of that, if you really look at it, there is Responsible AI, you know, Explainable AI, because in general, humans are forgiving when humans make mistakes, but they’re not forgiving when machines make mistake. So the question becomes, how do I explain what decisions I took? What is the reasoning behind what I took? And put into this your human-centric: Am I doing it for social goodness? am I adding transparency into it? What is the reasoning behind it? Is there trust in it? Are there biases in it? Is it fair enough.  All that leads into your Explainable AI and responsible AI or ethical AI, that is going to make sure that we are using AI for the goodness of the society, there is meaning behind it, as opposed to creating fake news, changing the personas of what people believe in and you know. Because with social media, it’s very, very easy to influence young minds, but how do you make sure that they know what is right, what is wrong? How do you make sure that it is responsible. So those are the things that are going to become very critical. 

And then the next thing is also cognitive AI, right? Context, reasoning, human behavior, why I did what I did. So all these are some things that people are talking about in AI, and it is going to evolve as we speak. But it is up to anybody’s imagination how the AI landscape is going to change. It is a fascinating world.  And to your point, there’s a lot of hype, there’s a lot of real world applications; we just need to see how much of it is research and stays on the research shelf as opposed to how much of it is really going to add business value to customers and is going to get deployed. So it is going to be an interesting balance of how it gets used, where it gets deployed, what is the real value behind all of it. 

But having said that, it is a fascinating field. It is a fascinating field, and it’s only going to evolve as we speak in the next five to 10 years. So can’t wait to see how this whole landscape is going to shape up.

[00:12:58] Camille Morhardt:  Kavitha Prasad, head of AI Execution and Strategy at Intel. Thank you so much for joining Tom and me.

[00:13:04] Kavitha Prasad:  Thank you so much, Tom and Camille.  Thanks for having me.  Really appreciate it.