Have you guys enjoyed your break? I spent a lot of time thinking about what it means to be constantly connected, and how virtual reality and augmented reality in particular changed how we experienced real and digital worlds VR, which lets you immerse yourself completely in a new environment. And AR which makes us digital images with the real world around you are still in the early stages, but are already being used for everything from entertainment to education, there's gonna be a lot more to come in the years ahead, as the hardware that powers these experiences gets better and cheaper, and creators make more content. Our next two Tr 35. honorees have fascinating insights about these technologies. We'll hear first from Jessica Bernhardt via video.
Virtual Reality is hard. It's been especially hard for those of us with filmmaking backgrounds. One because there's this expectation that filmmakers will know what to do with VR, which is often not the case, you do need to have a particular mind for this kind of thing. Second, a lot of the sure things in filmmaking like crafting a frame have become fairly elusive. The frame now has a mind of its own, it's reacted very much alive. And that's because the person in the experience driving that frame is reactive, and very much alive. I'm Jessica Bernhardt. I'm a virtual reality creator. And it's an honor to be recognized by MIT Technology Review. As a pioneer in the field of VR. I used to be a filmmaker at Google. I was there for eight years. And the last two years I worked there, I was their principal filmmaker for virtual reality, where I helped to develop what is now known as Google jump. I made a slew of different experiences that not only tested the technology, but also helped to get a better understanding of how a filmmaker might work in this space. These experiences touched on things like editing and montage. Being inside the mind of someone else. Collaborating with convolutional neural nets, and the weather channel. By exploring these things critically, I was able to hone in more on what VR is language could be. In a lot of ways. This comes from me taking what had worked for me as a filmmaker, and throwing it at VR to see what would happen. More often than not, it wouldn't work. But instead, something else would be revealed. Take a frame, one might think that since the frame is now controlled by somebody else, the frame just doesn't exist at all. But that's not the case. The frame is now a matter of human behavior
of probability.
What we're doing as creators is crafting premeditated worlds made up of potential frames. And as this is based on how we experience things, normally, there could be more than one frame, or there could be no way to predict the frame. And instead, the frame exists everywhere and nowhere at all. We can leverage these possibilities the same way we leverage close ups, medium shots and wide shots. And cinema is not just about the worlds themselves, though. Every time a person goes into VR, she becomes part of the chemical makeup of that experience. She has a mind that thinks she has a body that moves. She comes with her own set of experiences, memories and dreams. Possible VR happens when the worlds we create in the visitors to those worlds Connect. I put on a headset, and I feel like I'm there. But that's not enough. That's just surface. Great VR happens when the world and the visitor engaged with one another. My job as a creator in VR is to craft for the nuances of that engagement. We don't just edit because editing is fun to do, remove the camera, or have actors say lines or light things a certain way because we need to do those things. We created those processes because they were the ones that helped us create a great film. VR is hard. Identifying language for VR is arguably harder. But it's important. Because that's what's gonna allow us to come up with the processes that make great VR. making great VR isn't so that we can justify the technology. As you all know that stuff changes all the time. It's to really figure out why VR is important to us where it can go, and why it will matter to us in the future.
And I look forward to helping to figure that out. So thanks again to MIT Technology Review, to M tech for this virtual platform. And to all of you for listening.
augmented reality or AR technologies overlay, digitally generated haptic visual and audio feedback on our perception of the physical world. These technologies have been a dream since at least the 1960s. And they're just now on the cusp of commercial viability. Imagine that you're going through your daily life wearing a future generation AR headset, it can recognize people at the conference and remind you of their names, guide you as you're cooking dinner, or fixing your sink, overlay directions directly on the road. And of course, let you play Pokemon with your kids or by yourself. So in short, these technologies have the potential to fundamentally transform how we interact with and perceive the physical and the digital worlds. But there's also a potential dark side. The same immersive potential that makes AR so exciting, can also make it dangerous. Imagine now that you accidentally install a bad application from the augmented reality app store an application that's explicitly malicious or even just buggy. That application might block your view of oncoming cars as you're crossing the street, or plaster everything with distracting advertisements, or startle you with a screen spider. Or you might find it creepy that the application or your platform can constantly record and analyze everything that you're doing. And not to mention that everybody else's devices are doing the same as well. My work addresses a critical gap with emerging augmented reality technologies, protecting the security, privacy and safety of end users. Since 2011, when these technologies were still much more in the realm of science fiction, my collaborators and I have been working to understand and address this gap. Specifically, the question that my work asks is how should we design future AR systems to enable both these exciting and useful functionalities that we're dreaming of, while also protecting end users from the things that might go wrong? designing secure systems requires first identifying potential risks and vulnerabilities. So to that end, my work has systematically identified and classified vulnerability classes for AR including privacy risks from continuous sensor data collection, and security and safety risks from virtual content. My team and I then work to design new AR platforms that can mitigate these types of risks. So for instance, here I'm showing a simulation of our AR prototype platform that can enforce safety policies on the virtual content that applications try to display. We can prevent applications from blocking the drivers view of pedestrians, or ensure that virtual advertisements are only ever displayed on top of real world billboards, and nowhere else. many challenges remain to be addressed in this space, but it's critical that we ask and start answering these questions now before it's too late. While these technologies are still not yet widely deployed, and their designs not yet set in stone, my work lays the foundation for to enable these technologies to reach their full potential unhampered by security, privacy and safety risks. I'm Francisco rosener. And I'm an assistant professor in the Paul G. Allen School of Computer Science and Engineering at the University of Washington. Thank you.
Thank you. That was really fascinating. I've got a whole bunch of questions I want to ask you. Just one second, if I can get my questions. So let's talk a little bit about design. What do you think is some really bad AR design you've seen so far? what's what's an example of something really bad?
It's a good question. And I'm not sure. So I think it's actually kind of a hard question to answer because I think the technologies are still so emerging in their development, that, you know, these platforms aren't yet really commercially available. They are but the more sophisticated ones are, you know, like Microsoft HoloLens, for example, are available only in Developer Edition. So they're certainly more widely available AR in terms of smartphones and so on. And certainly there we've seen some tensions like some of you have probably heard some of the stories that happened with Pokemon Go where people maybe were not paying enough attention to the physical world. You know, or ended up in places that they shouldn't have ended up because they were so intent on their Pokemon? And I don't know that that you know that I would call that a design failure necessarily. I think it's it's a matter of balancing these various different tensions. And so for the more immersive technologies, like the headsets that are developing, I think it's actually still too early to really say, because there's a lot of innovation on the technology itself, and really enabling those exciting use cases. So now is really the time to think about what could go wrong.
Okay. So one thing I was thinking about when I was listening to you speak, and while I've looked at your researches, okay, is it really that likely that someone would choose to mess with me while I'm driving, or walking and using AR technology, such as one an example that we've talked about before is I'm maybe, let's say, 10 years from now I have an air windshield on my car, I'm driving along, and I have my directions overlaid in front of me. And then they see virtual people pop out at me or spiders or something like that? I mean, is that really all that likely? It's a good question. I
think, you know, certainly there are in every domain of technology in human society, there are always cases where people have adversarial intentions for various reasons. And one could speculate about those. But you know, one of the things that we think about also is that it's not just about intentionally malicious, malicious adversaries, it's also just about bugs. Even if you say, for example, you know, the augmented reality windshield application should only be written by a trusted developer, it seems ridiculous to have, you know, an open App Store for that sort of thing. Even if you say that, you know, you still want to be able to predict and or control what those applications may do when they start interacting with each other and unexpected real world situations. So even if it's not malicious intent, I think we need to think about how things might go wrong just on accident.
Okay. The academic an author, CNET Zeynep, tufekci recently tweeted that users should have hardware switches to give them control over whether their smartphones microphones are on or off. Do you think we need this kind of control over mics and cameras, when it comes to augmented reality, a lot of the headsets they need cameras in order to operate, of course, and a lot of them have or will have microphones as well?
Yeah, so I think there's actually two things here. One is on the sensor and data collection side, certainly, there's a tension because the AR, as you say, the air applications really fundamentally need access to all the sensor data to do the interesting things that they're trying to do. But there may also be situations, locations, or people around which recording is not appropriate. And so actually, some of the work we've done in the past has looked at trying to make that process a little bit more automatic, rather than requiring and users to constantly think like, Is this an appropriate time to record does, right? This person, you know, this person doesn't like my recording device, I need to toggle it. But it is an inherent tension with the functionality. The other thing that your question made me think of is not just on the input side, but on the output side, do we want a switch to be able to return back to reality? And that is
to real reality or real reality?
Well, we can get philosophical about what is real. But I think that seems like something worth exploring. But I think even that is challenging, because people will become dependent on these types of devices. And there may also be, you know, longer term effects on people's brains or perception and so on that just simply turning it off, can itself be debilitating? So I think there's a lot of challenges here.
Can you give us any examples of good AR design, either real or hypothetical? Even that you've seen out there thus far?
That's a great question. Thank you. Nothing that comes to mind off the top of my head, I guess, again, I think, you know, it's a matter of the fact that it which is not to say that it's all bad, but rather that the the use cases are still so exploratory sure that we've not really come to that point, you know, I think it is in a company's best interests. And I think they are starting to think about these issues, especially this interaction with the physical world can lead to harms that are much more tangible than maybe some of the harms we thought about only in the digital world. So I do see companies starting to think about these issues, which is great.
Okay, one, one last question for you. And we've seen with Pokemon, a lot of kids are interested in that kind of technology, or that kind of app in particular and AR technology in general. Do you have any thoughts at this point about using this kind of technology with children? It's just something I've been thinking a lot about when I think about children technology and how a lot of a lot of what's going to happen with that is really unexplored at this point.
Yeah, I think there's probably a lot of potential there you know, for education or other Or even just entertainment, other use cases. And this is not an area that I'm an expert in. But I, I imagined that it will be important to study kind of the effects on brain development and so on the same way that we worry about just exposure to screens in general exposure to constantly augmented reality, and and the things we might rely on there and how that affects brain development and children, I think, is certainly a question we should be asking.
Okay, and what happens when we turn it off? Yeah. We have time for a couple audience questions. If anybody has a question, we have runners that are becoming around with microphones. So just hold tight for one second. We've got one here in the front.
Raj mediatory. Our basic question someone new to the field, and there's so much jargon around augmented reality, virtual mixed reality. augmented virtuality just help us understand what that is, and which is the area that seems to be catching on emerging?
Yeah, absolutely, I think I typically use the term is virtual and augmented reality on one ends of on two ends of the spectrum. So on the one hand, you have, well, I guess reality and virtual reality spectrum, virtual reality, I typically think of systems that do not expose you to the real world at all, now, maybe that you're in a physical space, and it needs to take that into account. So you don't run into walls, but everything you see is virtual. Most of my work has focused on augmented reality where you still are seeing and interacting with the real world that is augmented in some way. But of course, that's a continuum and augmented reality itself can be implemented in a video see, through way where everything you know, you essentially are not seeing real light, you know, in the real world, but rather everything is, is digital. And so I, you know, I, I'm not sure that there's is clean cut breakdowns for this terminology, I think different technologies have different elements of it. And certainly, the the security and privacy issues, span that whole space, I've been most interested in the augmented reality side of things because of this interaction with the physical world that VR in the extreme is taking you out of that. But the vision for AR is that we continue interacting with each other with the physical world, we walk around, and we drive. And so the potential for physical harm, among other things, there is very real. We
have time for one more question. There's one right here and sort of center side over there. I think we got a mic.
I it's David CHATZKY with Deloitte. I was just wondering if you are aware of a commercial interest in this topic, their applications in automotive and aerospace and other so are you talking to companies that are implementing this or exploring it for commercial use?
Yes, I think I mean, at the highest level, companies are certainly interested in AR itself and all the promises it has. But you know, we've had some conversations with companies and other things I've seen, I think there is a growing interest in the safety side of things. Because for various reasons, one is just that's that's the right thing to do. But also, of course, reputation and, and other aspects. And again, this kind of interaction with the physical world is really critical here. It's other types of issues we might think about in computer security and privacy are a little bit more abstract. We can have conversations about privacy, and we can argue about whether we actually agree that some of those things are a problem. But if you're a company making an AR windshield, you do not want to have the story that says there was a crash because of something that went wrong. So I think I think there's a lot of incentives there.
frenzy. Thank you so much. It was so good to speak with you.
