[00:00:37] Camille Morhardt: Welcome to Cyber Security Inside, and this episode of What That Means. We are going to talk about intelligent systems today, and that’s all in the context of artificial intelligence and augmenting human experience.
I have with me today, Lama Nachman, who is a fellow at Intel, and she’s in charge of the relatively newly created Intelligent Systems Research Lab within Intel labs. It’s actually a combination of several other labs that have existed for a long time, so she’s been doing research in this field for quite a while, and we’re going to ask her what her lab does, and what intelligence systems mean? This is Part One. We have a Part Two where we speak with three of the people who work and run research projects within her lab. Welcome Lama.
[00:01:28] Lama Nachman: Thank you so much, I’m really happy to be here. So Intelligent Systems Research Lab, basically we are a multi-disciplinary lab that brings together social science, design, AI design, and hardware, and software engineering. What we’re trying to do is figure out how do you augment and amplify human capabilities and experiences with AI. We use a social science based approach to understand what are all of these unmet needs in the world, and then understand what type of innovation needs to happen on the AI side to actually bring major improvement in these life experiences for people. That’s the experience level and a lot of the human AI collaboration pieces of that puzzle.
We also do a lot of innovation at the lower level on the algorithmic side to try to in some sense solve some of these hard problems that today are standing in the way of scaling AI in real world applications that range from how do you improve accuracy? and learning in new environments that it hasn’t seen before? How do you actually do inference at extremely large scale? learn from very limited data? And then try to get to things such as explainability and usability of these systems.
[00:02:52] Camille Morhardt: Lama, can you tell us what research you personally have been focused on over the last couple of decades?
[00:03:01] Lama Nachman: I came to Intel in 2003, about 18 years ago. My work actually started in wireless sensor networks and trying to connect the physical and digital world. A lot of sensing and understanding from sensed data to try to change the way we work in different things. We move towards trying to sense people and apply a lot of that towards health applications. Prediabetes monitoring is an example, then we moved on to scale understanding context in general and that started with a lot of on body sensing. We’ve done a lot of work with wearables and things like that and then we expand it towards understanding essentially everything that’s happening in the environment that I don’t do. The whole notion is that if the technology is able to understand people and what they’re trying to do in the world, not necessarily just from interacting with the PC or interacting with the phone, but actually what you’re trying to accomplish in your day-to-day life, in terms of learning in a physical environment or working in a fab or just helping people with disabilities, as an example, what can we utilize as signals in the physical world? Then with a lot of algorithmic innovation, turn that into understanding so that we can better facilitate experiences for people as they traverse their normal life.
[00:04:32] Camille Morhardt: Is this getting smaller and smaller? Like from on-body to in-body? Are we seeing a trajectory in that direction?
[00:04:39] Lama Nachman: We see smaller and smaller and we see bigger and bigger, so actually we see a reversal in both of these spaces. Sensing has gotten smaller, computation has gotten much more efficient, so that we’re able to actually materialize systems and tiny, tiny implementations. It really is spreading into the environment in ways that we have not seen before. So if you think about it in terms of data, really it’s translating an unbelievable amount of data.
[00:05:10] Camille Morhardt: But you’re looking at augmenting human experience, so we’re focused on humans here and using technology and using sensors to understand better what is the human experiencing and then improve that experience, is that kind of the high level?
[00:05:27] Lama Nachman: Yeah. A subtle point of distinction is that when we talk about this, because there’s a lot of work that you can think about in terms of how do you improve the experience with technology, as in I’m using my PC and I can make that experience better, in some cases we do that, right? Because the PC ends up being, for example, the conduit of communication. Like right now you and I are communicating. One of the things that we know. People struggled with throughout the pandemic is what we call the Zoom fatigue, right? Or any telepresence fatigue and that’s because there aren’t really, I mean, trying to bring that back to 2-D communication is not something that actually makes sense. People use all of these subtle cues and actually better representation of the physical world is much better to enable that type of communication. So in that specific case you can think about it as it’s enhancing that experience that’s facilitated by the technology.
But if you think more broadly, when you step into a manufacturing setting and you have a technician who’s actually working on some equipment in the fab and they’re trying to fix something or transition from one task to another–to print a totally different processor or something like that–they do a lot of these tasks in the physical environment. And a lot of the work that we do is bringing technology to understand what people are doing in the physical environment and then assist them, right? So if you understood what somebody is doing and if you understood what is supposed to happen and the AI system can actually converse well with the human, then you could see how you can start to think of these things as human-AI systems, where we’re bringing the best of the human and the best of your AI system without getting stuck at what’s not good about either one of those. Right?
We’re trying to really leverage the diversity in human-AI systems to say “people are good at some things that AI is terrible at and vice versa.” R ather than get into this place where we’re trying to replicate humans, how do you get into this place where you can actually make AI systems much more resilient, make them shine in the things that they really do well, which is processing massive data and finding patterns and massive data, but humans are not really good at that. But they’re very good at learning from limited data in totally ambiguous situations that they have never seen before. An AI system struggled with that all day long. Both of these things are usually needed to solve any problem in the real world, so that’s what we’re really trying to do.
[00:07:59] Camille Morhardt: Give us an example of something, it can be far-fetched and it can be hypothetical and maybe not happen, but what sort of thing might you imagine this could enable in the future? What kind of a use case or scenario.
[00:08:13] Lama Nachman: Let me give you a couple of examples that we’ve been working on, one is in a manufacturing setting. It’s amazing because we’ve done tons of automation in general, especially in chip manufacturing, but you walk into a fab and you still see tons of people. It’s not that people disappear, they just do different tasks in the fab. These type of tasks have specific set of specs associated with them. You could imagine having an AI system that can ingest all of these specs through natural language processing and understand that these are the different tasks that someone should be doing to actually perform the specific procedure.
Now you could train, and this is what we’re working on, train an AI system to recognize the specific tasks and sub-tasks to actually aid people in figuring out what information they need in the moment, what specific next step, sometimes people forget things, right? And those are the things that if they, some can take all of this natural language processing and bring in the looking through the camera to understand physical tasks that people can do and make that link, it can then massively bring a lot of information, including information that it has seen across multiple users.
So enable human to human learning. Because what you see in these settings is people who have done this and they’re experts and you have novice people and they don’t know these things and sometimes experts skip things for whatever reason. So having that ability to dialogue with the person and highlight these things and have the person help the AI system in it’s learning.
There’s like, okay, no, it made a mistake in recognizing this thing. And the person can say, no, actually this is x. And then the AI system continues continuous learn. So now you’re bringing in some of the knowledge of the human specialty of the experts and helping train the AI system and what should happen.
Another example that we’ve been working on quite a bit is actually in early childhood learning. And there we’ve done a lot of ethnographic research that pointed to parents having this dilemma: on one hand they really want to bring the best that technology can bring to their child’s education because they see the benefit of that and they’re extremely worried about screen time and rightfully so. We’ve seen a lot of the research on how that is actually translating to a lot of issues in the longer term.
If you step back and think about it, how does learning happen in early childhood? If you walk into a kindergarten class, you don’t see people on laptops and tablets, right? You see children on the floor, on the rug playing with manipulatives, learning math, by putting objects together, all these types of things happen in an actual classroom environment. One of the questions that we were asking is how can you get an AI system to observe what’s happening in the room and be kind of a peer learner? If it can help a student, for example, in math learning, maybe they’re putting blocks and blocks together. They’re mapping to tens and ones and additions and subtraction and comparisons and things like that, but they’re doing that in the physical world. So that means that technology needs to come into the physical world, observe them, and then have a conversation with them that is actually situated in that physical world, because that’s what it needs to be able to do.
[00:11:30] Camille Morhardt: Where is this compute or this AI going to sit? It’s interpreting natural language. It’s interpreting video or gesture or different cues that humans give. Where is it? Are we talking to great eye in the sky now wherever we are?
[00:11:46] Lama Nachman: If you turn the whole environment into a smart environment, the other part that I haven’t talked about is ultimately what you’re trying to do is bring engagement. Because what we know from pedagogy research is that learning outcomes correlate with engagement of students. So to try to get this to be an engaging environment we’ve actually gone all the way out. So this thing actually projects, this peer learner, he’s aware of the physical space is a teddy bear like thing that jumps all over things that kids can help him build structures for him so that he can traverse it, and because of the fact that we’re able to map the environment, we will understand there’s a box that somebody put in, but some that Oscar can jump on and land on and still obey the laws of physics, but it becomes a kind of digital-physical world connection that makes it much more engaging.
[00:12:32] Camille Morhardt: Is this actually a robot? I don’t knowe that I understand.
[00:12:36] Lama Nachman: No, it’s actually projected, it’s a projection space, but we’re able to map out the physical environment. Imagine you have this thing on the wall that’s moving, and then there’s the box the kids have put up, right? Then he can jump and land on that box. So almost see him bending out of the wall over the projection onto that physical, so it becomes much more engaging and we’ve done deployments in the school where we could see a very different level of engagement because of the fact that now they are engaging with Oscar, as if it’s actually a physical thing in that world, even though it’s not actually embodied in like a robot.
But to answer your question about where does it live? Just imagine you have a huge wall with projected everything. You have lidars, five cameras in the environment, and things like that. This is turning a space into a smart space. Now you have a server running in the school that’s actually doing that.
That is great, if you’re actually trying to understand what is the unlimited version of that, one of the things that we’ve done, especially when COVID hit, because this project actually started pre-COVID and then COVID hit right in the middle of this project. I managed to get one deployment in a school in Hillsboro before that happened and that gave us a lot of insight.
Then we said, “what is a minimum viable thing of that experience that you can actually deploy in people’s homes?” That would be something that we’d need as well. Then what we turned towards is trying to bring that Oscar character and have not a huge physical environment, but think about a rug on the floor with a laptop that facilitates that sensing and computation and all of that stuff.
You’re essentially still creating a smart environment because the kids are sitting on the rug and playing with these manipulatives, but it’s actually enabled by a laptop and a couple of peripherals. And that’s clearly one of the things that’s important for us, things like privacy and so on. We make sure that all of that processing is actually happening at the front end and so on and so forth.
[00:14:41] Camille Morhardt: This is kind of like taking the digital world and using it to enhance the real world. I can’t help but think about all the conversation around metaverse which is basically moving people into the digital world or the concept is that. And how would this technology play out or become a catalyst for something like that?
[00:15:06] Lama Nachman: If you talk about metaverse and it’s funny, cause I’ve talked to multiple people and friends that I have at Facebook and other places, and everyone has their own interpretation about metaverse. So sometimes we talk about it as if it’s a known thing-
[00:15:18] Camille Morhardt: Well, maybe you should actually define it quickly.
[00:15:21] Lama Nachman: Its about everything is becoming ubiquitous. You’re actually connecting the physical world and the digital world in some interesting ways. You interpret things differently, you have all sorts of different devices that have to live within that that makes these types of experiences possible.
I want to step back for a second and say, ultimately there are all sorts of experiences within that spectrum–from total virtual and reality where everything is virtual to everything analog and everything in between within that spectrum. Right? To me, what’s really interesting about this is what is the problem that you’re trying to solve and what are the concerns that you have that you’re trying to mitigate.
When it comes to early childhood learning, what we’ve heard time and again from parents is that they really want this to be as analog as possible with intelligence just enabling or bridging the gap of where the analog equivalent is not possible. If you were to have a teacher for every child in the world that would be fascinating, but that’s not reality. Then the question is how do you not then turn that into the other extreme, which is now everything is virtual. Like if everything is virtual, it’s simplifies how you consume content? how you deal with it? If you start to say that “no, I just didn’t want the physicality of the physical environment, but I want actually the intelligence to comprehend that” it’s so much harder problem, right? Because if I put you in the computer, it’s much easier for me to actually do everything and observe everything that you do. And I think that is really the struggle because in some things, totally virtual is absolutely fantastic–especially if you’re enabling things I want to do remote health work, right? That’s fantastic. It works out really nicely and it does everything that you need. We have to understand that there are very different experiences within that spectrum that needs to be enabled, and technology needs to come in to to understand where it plays and where it doesn’t.
[00:17:30] Camille Morhardt: And you work with contextually aware and multimodal. Can you talk about what are the different kinds of things via wearable, via video, or via whatever sensors you’re using to actually get a sense of the environment?
[00:17:45] Lama Nachman: We essentially utilize visual input. So camera images or video, these things are very different, right? Sometimes what you’re trying to do is get to action recognition and you really need a lot of the details. Sometimes we’re trying to understand activity, such as a kid is actually jumping up and down and doing these type of things.
We do a lot of work with audio and with audio, I mean both understanding background audio, what’s happening in the background—so like the scene of the environment and that provides a lot of context. What you’re hearing in the background and what you’re hearing in the foreground, which is actually the speech, right?
We do a lot of work with text. We do a lot of work with physiology, so things that have to do with emotion and engagement and things like that have a physiological component. If you’re able to detect things like heart rate and skin conductance that gives you a lot of input. We also do a lot of work on BCI (brain computer interface). So we get EEG signals as well, especially for people with needs for accessibility. Essentially you essentially wear a headset that has electrodes and then what we can do is capture EEG or brainwaves to enable people to communicate with the PC because they can’t move any other muscles. Imagine someone who’s totally locked in, such as someone with ALS, that can’t move any muscles. We’ve also used all sorts of other signals to capture muscle movement for people with a disability. If we’re talking about more of the mainstream sensing.
We also have been doing a lot of interesting work using Wi-Fi as a sensor. And that might not be obvious because we think of wifi as a method for communication. But if you think about that–specifically when you get to the issue of privacy–when we move around in the physical environment, we disrupt Wifi signals and wireless generally. So we’ve been trying to utilize that disruption instead of being a noise on communication, being the signal of human activity.
We try to essentially take that data and turn it with machine learning algorithms to understand people’s activities–what they’re actually doing in the physical environment. The cool thing about this is that from the ethnographic research we’ve done with elderly people in their homes, they have a major concern about having a camera watching them–even though they’re willing to have a system that can understand that they’re doing fine and they’re progressing usually with day-to-day activities because then they can stay longer in their homes; they’re fine with that inference. They don’t want a camera because they’re worried about somebody seeing them in all those settings.
We try to ask the question what if you actually to solve some of the privacy issues, one of the things that you could do is reduce the gap between what is being sensed and what’s being inferred. With wireless sensing you would actually sensing what you can infer, right? You don’t have all of this high resolution images and things like that to then infer that somebody just went to the kitchen to do something. So that that’s another interesting signal that we actually use.
[00:21:01] Camille Morhardt: How much can you tell about a person from their interference with a wifi signal? Are we talking about where they’re located or are there other kinds of reactions?
[00:21:15] Lama Nachman: It’s a lot of things on a whole roadmap and a lot of complexity over time that gets uncovered as wireless signals get better, as you can tap into more things from the platform, as computation getting better and things like that. We can detect a lot of things if it’s the same environment tand you can take out some of the changes in the environment, out of this picture; but to a large extent, one obvious thing that you detecting is motion. When you detect motion you can start to look at what type of motion maps to what activity. One simple thing you could detect is if somebody is actually walking towards their PC to turn it on, and then you can start turning things on in the background to make it much less more timely to get to it once you get through it. You can understand that somebody is actually walking around in the kitchen. Over time you can start to understand this looks like somebody is preparing dinner.
Another area that we’ve been focused on is actually getting physiological segments, so you can detect breathing from wireless signals because essentially what’s happening is that your chest is moving. On one hand, you could argue that movement is much smaller, so it’s harder to detect. On the other hand, it’s much more periodic. In some sense, there’s something easier about detecting something like that. Today we have some POCs that are showing the texting actually breathing just from wireless signals.
[00:22:35] Camille Morhardt: Let me go back to one thing you said that I found really interesting. You were talking about patients with ALS or people who can’t move even to signal yes or no. I know a long time ago we could tell people to picture playing tennis or picture walking around their room and one meant yes, and one meant no, and therefore we could communicate with people that we had considered vegetables and it’s come a long way. So can you tell us how far it’s come?
[00:23:02] Lama Nachman: I would separate that into a couple of areas. The things I was referring to active communication–not necessarily understanding thought, but actually understand that if you have the intention of doing something and the way it works is imagine, let’s say you have a lot of different options on the screen or a keyboard. You flash different things on the keyboard. If you would intending to type a specific letter, when that letter flashes, we know that there is a different signal that comes out in the EEG that we can tap into. It’s very noisy, it’s not very clear, especially that this is not embedded in your brain. This is just a cap that we are wearing. That makes the fidelity of the signal much less. So then if you do that, then basically what you are able to do is with multiple repetitions to reduce the error and we’re bring in language models into the picture. Then you can start to guess, “Oh, this person meant to type this letter.” That is what we call the BCI, which is brain-computer interface. You’re trying to interface through that brain computer system.
The second thing that you mentioned is a lot of work that has been done—and in fact Ted Willkie in Intel labs in collaboration with the universities he’s done a lot of work to try things like FMRI and looking at images to try amd understand can we in some sense classify thought, if you would write through these methods? That really requires imaging the brain. You’re looking at data from a lot of people and trying to understand are there specific patterns that you see. I frankly don’t know a lot about this area I haven’t done a lot of work in it.
In some sense, what you’re doing there isn’t a controlled thing, When you’re trying to interact, I give you a keyboard and I’m going to flash things in a way that I can make my detection easier because ultimately the goal is actually communication, as opposed to more generally trying to understand thought that’s more implicit.
[00:25:07] Camille Morhardt: Free flowing and unstructured data, but if you couple that with the physiological reactions you could get a lot closer. Interesting. A lot of things bring up privacy concerns and ethics concerns for people. A final question, can you let us know what your lab is thinking about with respect to this or how you can take in those considerations?
[00:25:40] Lama Nachman: Totally. One of the really good things about looking at this from a multidisciplinary perspective is that much earlier in the cycle, and it’s actually wsomething we’ve been doing all across Intel which is how do you enable responsible development of AI? And that means at the very early stages you’re asking questions about risk. You’re looking at the project as a whole even before you start developing. Because If you think about ethical concerns, these concerns happen at different stages. There are ethical concerns with what a system does and whether it should do this in the first place, which is really kind of what we try to flush out early on.
Then there are ethical concerns about, is the thing that I developed, in the way that I’ve developed it, starting to bring ethical concerns? So the example that I talked about with wireless, for example, versus camera is one about the development.
We take this approach on pretty much everything that we do, right? We’re looking at the risk for the specific problem and say, “what’s the least amount of data that should be captured?” How do we infer as close to the sensing as possible? so that data is reduced and you’re not sending raw data back which then makes the privacy concern more, right? With the whole human AI collaboration piece, we’re trying to understand how do you actually bring the human capability into this picture rather than assume that the only way that you can actually bring AI and innovation is to replace humans. How do you bring better outcomes when you think about the key role that humans play into the puzzle?
We look a lot at equity, right? A lot of the work that we do with accessibility is really rooted at how do we bring equitable outcomes, despite the range of abilities and capabilities that people have? So that’s a big part of what we do. And then of course, looking at bias in a lot of algorithms that we build and ensuring that we’re not replaying the bias that’s in the data in the world back at us through the AI systems that we built.
So, those are some examples that permeates through everything that we do within, within the lab.
[00:27:44] Camille Morhardt: Super fascinating. Lama Nachman, Intel fellow running the intelligent systems research lab within Intel labs, fabulous conversation. In a part two, we’re going to talk with several members of your team and dive a little deeper into the specific kinds of research that they’re looking at withi n your lab. Thanks for joining us.
[00:28:03] Lama Nachman: Thank you.
