The AR Show: Ross Finman (Augmodo) on the Approach to the Selling 1 Million Headsets (Part 1)

11:47PM Oct 2, 2023

Speakers:

Jason McDowall

Ross Finman

Keywords:

niantic

ar

gaming

company

applications

thinking

year

talking

world

acquisition

headset

experience

form factor

building

kid

self driving car

tech

idea

fundamentally

device

Welcome to the AR show where I dive deep into augmented reality with a focus on the technology, the use cases and the people behind them. I'm your host Jason McDowall.

Today's conversation is with Ross Finman. Ross is the CEO of Augmodo, a company utilizing AI and AR to revolutionize grocery store experiences and economics starting with the personal shoppers who fulfill online grocery purchases.

Previously, Ross spent four and a half years at Niantic. There he founded the AR mapping and visual Positioning System effort before becoming the AR strategy lead, and then the general manager for the AR headset group. He joined Niantic through the acquisition of his first startup Escher Reality which became the foundation for Niantic AR platform now called Lightship. He started astral reality as he was finishing his PhD in robotics from MIT after previously completing his undergrad in computer engineering from Carnegie Mellon.

In this conversation, the first of two parts, Ross describes his perspective on the AR market opportunity. Specifically, we get into his journey at Escher Reality how it started a couple of key moments during the company's early days, and the sale process to Niantic and some lessons learned along the way. He described some reflections on his time at Niantic, including the open mindset they had about developing new types of hardware. And he shares his current perspective on where AR has the most immediate potential, but come back in part two with a discussion about his current startup Augmodo. As a reminder, you can find the show notes for this and other episodes at our website de ar show.com. And please support the podcast@patreon.com slash the AR show. Let's dive in.

Ross, it's so so happy to be able to chat with you again here on the podcast. We've had a chance in the previous couple of episodes that we've done. While you're at Niantic to talk a bit about your background, including growing up on a llama farm, which is a fantastic story and worth going back and listening to for those listeners that haven't heard it.

MIT people dime a dozen, not too many llama farmers.

That's true. But one of the things that's new in your life that I wanted to come back to here is that reflecting back, you did all this PhD work there when you were at MIT. And you spent all this time working in AR and in robotics, and in building out these AI models and trying to teach some sort of system and artificial system how to understand and navigate the world. But now you're a new father, and you have a toddler running around the house. How does being a father to a young child compare?

Well, I also have another kid on the way into three weeks, I think when this comes out. So yeah, gonna have a double duty but fantastic question. It's really like being a father. It's an amazing experience. Being a roboticist who's a father, it's actually really disheartening, just in the sense of you can spend, there's some robots that are out there where people spend 510 plus years, trying to get them to do things. And then you can have the Boston dynamic style robots that can do backflips, they seem like they can do anything, but just look on a lot of their systems, they have QR codes on there, because solving the perception problem. And you realize that in mind of a non potty trained two year old that's running around the house, like wanting to put everything in its mouth and eat it, realize, wow, that's more sophisticated than most algorithms that some of the top people in the world have ever built. It's quite a humbling experience. But I would say the just the idea of being a father, and you're kind of just a classifier for like your kids, you're like, yes, you eat broccoli, yes. Put on your shoes, like no, don't put that in your ear. So you're constantly like just being a data labeler for your kids, perhaps don't have a CPA showing up. My daughter's saying that but like, it's kind of a just a mindset that you can have kind of putting on your academic hat as you think through like, oh, how would I train a robot to do this, you're like, Oh, my God, I'd have to have like a team of 20 people doing all the data labeling, that would take like three months work. And then finally, you get an enough data to then train it for that specific use case. Meanwhile, like my son, looking at some of these, like children's books, they're very abstract. And then you look at that they're like, Hey, what's that? You know, like, I think it's a giraffe could be like, probably not a tiger. But then he can kind of learn the words in a book. And then we go to the zoo for the first time. And he can make that and putting in AI terms transfer learning jump have like, this kind of small, cutesy abstract version of a giraffe. And then he looks at it, and you recognize it, having never seen a physical giraffe before. And now that's his favorite animal. But it's just absolutely amazing the amount of cognitive leaps that kids can make. And then it's like, Oh, and one week, he's counting from up to three. The next week, he's counting up to 20. And you're like, where did that come from? I only started practicing up to 10. Where did he get the other numbers? And then you're like, Okay, picked it up from just a couple examples on there. So it's a being a new father. It's just absolutely amazing how fast kids learn and how curious they are, and the big jumps that they can make and the speed at which it is when you think about some of the robotic stuff. You can have amazing videos, but it can take like, kind of months and months or even years to create. Like all the fundamental technology behind this videos, really just humbles you in terms of how So far AI and robotics have to go and just even unlock the concept. And not sure if you've heard of it and press too much of a side tangent, but embodied learning of like, humans can be their own feedback loop. While most AI and computer vision stacks, it's like, you collect the data, you manipulate, you train them models, everything else, and then you, and then you look at the data, and you get the outputs on that. And you have the results on there. And it's kind of a one way valve. But kids, it's like, hey, I want to learn are these two blocks connected? And then you can separate them? And then it's like, oh, no, they're not. And they can change. And they can be their own classifier on their own. And you as a father, it's just kind of there to add guardrails on top of it. But I could spend an entire podcast talking about everything I've learned from my own my own kid,

it's amazing how much they teach us in the learning process.

Yeah, and just they like, it's really you get to meet them. It's not like you get to mold them. That's probably one of the biggest changes for me. It's like, Nope, they're their own entity. They're learning at their own rate, they have their own personality, and then you just kind of pure along for the ride and make sure they don't like go face first off that to the playground. So,

so true. One of the other things that you had done when you're at MIT, and all the research is that you you saw an opportunity in the market, and you took the work that you were doing, and you created your first startup at the time as your reality I was back in 2016, can you take us back in time to that point and kind of share the opportunity that you saw emerging in the market then.

So that was an interesting time. So we started the company, my co founder and I old college friend, we've been talking about different startup ideas for a while. So in 2015, we're like, Hey, what are some different ideas, but really, from a computer vision standpoint, there's really three options, there was self driving cars, which were really hot at the time, everyone's like, oh, when three years you're gonna have self driving taxis going all over the country. Back in 2015, we're still waiting for that you had drones like that was during the hype cycle of drones of, hey, this, you're gonna have drones for everything. And now let's kind of dive down into well, I mean, still a good business. But it's not quite the hype that it was supposed to be. And then you had this kind of vague fuzzy AR VR where you had Magic Leap, raising billions of dollars at the time, Microsoft HoloLens, one was just kind of getting announced at that point. Or maybe this was a little bit before, probably the biggest passion, there was meta buying or Facebook at the time buying Oculus for, like billions of dollars. So then it was like a, from a self driving car perspective, I'd been in that space, my advisor ran the MIT self driving car team saw a lot of stuff in there, it's actually a great background, but it was like, I feel there were so many limitations on there. Like, fundamentally, it was a cliff problem, not a gradient problem. And what I mean by that is, hey, if you have a robot that works, or self driving car, that works 99% of the time, that means you die one out of every 100 miles. So you kind of need to have like seven nines of reliability on there, before we can launch it. Meanwhile, in for drones, there's already some good companies out there skydio, they were just starting at the time, spun out from a neighboring lab of mine at MIT great technology on there. To just help, there wasn't so much opportunity in the drone space. And I wasn't sure where it went more than like putting a camera someplace you want to see, which turns out is a multi billion dollar industry. And AR VR, honestly, just because it was all talked about heralded as the next future of computing. I didn't see a lot going on in there. But I saw a lot of money moving. So my co founder and I were talking through different ideas were like, Hey, we could probably do this on a phone and have like, the a lot of the Magic Leap style videos at the time are like, oh, let's have some applications on there. Maybe do a Tomodachi style game, which people are still talking about to this day. But yeah, there there's just a lot of promise and opportunity. And it seemed like more of an open canvas. While I knew all the ins and outs of the self driving car tech stack drone space seemed a little bit busy. It's kind of just what were the options that were out there. And my co founder and I, Diana, we got very excited about oh, and you can do this with any AR and this AR and it's one of these, as you probably know me just as much if not more than me. It's very addicting vision, once you start thinking about it, and then you kind of get like wrapped into it, oh, this would be great. And we thought, hey, if we can just get our little, little steps towards that. And at the time, first idea was how do you get AR on mobile devices? This was before AR kid even before Pokemon GO came out. And we thought, oh, as technologists, we make this one technical unlock all of these applications. And we learned a lot in the process.

So what is it the ultimately made while you were there? So the first step was can we get AR to work on the phone, absent these existing frameworks, you're building your own initial framework, I suppose, to enable the access of the sensors in order to deliver an experience. Is that the kind of step one?

Yeah, so the trajectory that we went through was building slam systems or insight of tracking on the phone and getting some of the initial applications on there. And we initially thought, or the origin of the story of the company, we were actually called AR spirits, because we thought the first application would be AR spirit animals, kind of Tamagotchis. And you can see how naive we were at the time, but our business model was, hey, this will be a social media play. And if you want to like hug someone else's spirit animal that's free, but if you want to flame thrower them with a torch, that'll be 10 cents. So we wanted to like charge cyber bullying, which is also a hot topic at the time. So if you could tag cyber bullying, we thought, oh, it's gonna make a lot of money. But then there we quickly ran into, we don't know how to make games or not gaming people, we don't know social media space, we were hardcore technologists. And once we start building up the technology and talking to other game studios for advice, they're like, Hey, this is kind of interesting. And then we pivoted quickly to just being a platform play. And initially, at that point, we were still doing like mobile augmented reality, this was right after PokemonGo came out that we focus just on the technology part. And then from there, we quickly ran into, like, you keep running out of memory on the device from the amount of space that you have. So if you're building up a map of a table, building up a map of a home for gamers to play in our park, like you quickly run out of what the device can do on its own. So then we're like, Okay, we're gonna have to offload some of this compute to the cloud. And that fundamentally turned into what issue reality did and our pitch line was the back end of AR, eventually, someone else coined the term, the AR cloud, after we launched southern that kind of took off, and probably many of your audience have, remember that, like everyone discussing the AR cloud, and there's kind of like one year where a bunch of companies started going after the space and, but really, their idea was, you need to understand reality in order to augment it. And that required a lot of mapping data that needs to be processed on the cloud offline, if you wanted to leave a Pikachu on a bench somewhere, you need to have a model of the bench, so you could come back to it. So that was the whole idea of persistence and multiplayer. And to my knowledge, we were the first or one of the first if not the first company, kind of actively building that stack.

Have you ever cornered met my snakes or or in bar in arm wrestled over, like, who adopted the concept in turn first,

oh, I fully give them the entire credit for getting the AR cloud and getting all the buzz bird, they may not want to hear this. But inside Azure, we talk to them as the external marketing arm of Escher reality, because they got so much interest and excitement about the AR cloud space. But if you look at like our Y Combinator kind of demo day, and our press release on there, and when we finished fundraising, it's about two or three weeks before they really started beating the drum, which for us, that was a fantastic time, because then you had your literally your three weeks off of raising your seed round, you're all excited, you have a couple million dollars in the bank. And you have big name VCs saying, Hey, do you want $10 million for series AIG? And it's kind of a the opposite of most founder problems. Like you're kind of sitting there thinking you're like, Okay, this seems too early. But if you don't take the money now, we'll spend two years and have a harder time fundraising at worse valuations. So then you're like, there's a strong incentive to think about taking the money at that point. At that time, and perhaps jumping a little ahead and story there was also a Niantic approached us. And we started having very positive conversations with them.

You ended up ultimately taking the path that Niantic offered as an entrepreneur, what was the debate going on for you? And among your, your you and your co founder, Diana? About the different options there? Like how did you evaluate?

So for any entrepreneur deciding an acquisition, it's a really personal decision, especially in the early stages, when there's so much promise for what you're going after? Because you can you have the money in the bank, it's not like, Oh, we're three months away from having to lay everyone off. So there's kind of a different mindset, if you're going for a strategic acquisition versus a necessary soft landing, or you built up kind of an initial business, and you have an idea of what the growth is. So it's a very, we were fortunate enough to be put in that position. But there you're just trading off, okay, you are technically giving over control over the direction of your company, will they be good stewards of your vision? Because fundamentally, everyone gets into entrepreneurship? Not necessarily, there's a desire to make money, but you do want to do it, because you believe, hey, this can be huge. You put your identity in there, and it's like, hey, will this new company like my baby? Will they take care of it? Will they nurture it? Will we still be a part of it? So you can think of it like being a getting back to the initial story of a parent and then you're getting your kid up for adoption, but then they're saying, No, you can come around and still be involved and help raise your kid and everything else. But fundamentally, it's not your kid anymore, but from a like just talking with them having meetings with the CEO John Hanke being like, okay, he knows the experience because his company was acquired by Google and he got to see that and we had one of our are champions Michael. He was like, Okay, here's everything that's going to happen, I'll be there through it all, he was very, very good at saying, No, we're believing in this vision, we're gonna go after it. And to their credit. Now when you think of Niantic, there's the gaming side. But then they're also invested a lot, we were the first augmented reality acquisition, they're really focused on building out the map of the world, enabling Earth as an AR game board, and even expanding beyond some gaming applications and testing things out. So to their credit, that was a significant transition in the hair history. So which many other companies like oh, what director level wanted certain headcount. And this was a political maneuver, and they pull it in there, and it ends up destroying the company. So while it was a no acquisition is just hey, you just show up to a different doorstep, and everything is the same as normal, but like, from a overall direction standpoint, was surprisingly aligned. A lot of our investors, they're like, Hey, we just put in money, why are you selling? Why are you selling now? And then they're like, oh, and then you got to hear all these horror stories of everything that all the ways acquisitions fail the lawsuits that enabled talking different founders saying this was the worst decision that I had. And while there were the bumps along the way, like, Niantic did want to go after that vision. So ended up

being a really great alignment. And they convinced you of that upfront, and you felt that it was going to be the right decision for you and your and your co founder.

Yeah, I mean, initially, we I didn't think that it would go through and there was one meeting that I had, where it was one on one dinner with me and the CEO. And afterwards I like called up to and I'm just like, I, I think the deal is dead. Like, it's not going anywhere. And like, this was a potentially waste of time, I'm sorry. And then they came back three days later, and it was like, Oh, really. So it's very emotional rollercoaster. And because everyone looks at my startup career, and you can say it was very quick. But I'd say you don't count the years off end of my life, I'm going to die at 82. And so 87 from all the stress, because like during that whole acquisition process, you're like, Okay, what are all these different legal terms mean? And then these are the experience big company people, what are they trying to pull on? They're like when you're at opposite sides of the table. So literally, I was sleeping about three or four hours at night, like a go to bed at midnight, just kind of working on stuff, thinking through it. Waking up from horrible nightmares at three or four in the morning, like, Oh, what is this mean? Is this a tactic that they're doing? And there's kind of this opposite table like, in hindsight, it was, it was the first major business deal of my life. So I was being a lot more paranoid. And when you're faced with a lot of uncertainty, there's always the boogeyman in there. I like to say we biologically evolved to think that rustling in the bushes is a saber toothed Tiger trying to kill us and not just a fuzzy bunny. And again, to Niantic credit, they weren't that experienced on the acquisition front. And they were like still figuring things out on their side, like we were their first major acquisition. And they only did one smaller Aqua hire before us that they had already had a like 510 year relationship with the founders on there. So they're like, oh, jeez, Ross is paranoid. And I'm like, Oh, they're trying to play all these games, but just open communication and talking through it. And then yeah, making us feel comfortable. And then feeling like it was good to give them a lot of credit for dealing with me during that time, because I had a lot of green lessons learned. Going through that process. So Vikram, if you're listening, thank you.

I think a lot of founders can appreciate the the sort of stress and paranoia that you were living through at that time, as you kind of reflect back because that was your first entrepreneur experience, you'd come out of, you know, out of Premier educational experience at MIT, you end up going to Y Combinator, you end up raising money through that end up having opportunities to choose between acquisition or additional investment or just staying on the path that you're on. And all these is a first time founder. So as you kind of reflect back on that, that first time founder experience, what were one or two of the kind of key lessons that you like to impart to other founders. After having gone through that,

just one or two, I'll try and summarize. Number one, people really don't care about tech. Like as a tech founder MIT, there's an ecosystem of like, all the tech is what unlocks everything. Everything's built off the tech is very engineering mindset. And that is true to a degree. But tech is also two steps away from a customer. We started the company in 2016. But actually view the first year, like we weren't full time on it. We were still figuring out a lot of the details on there. But it was still we were building Tech, we disappeared for six months had an advanced slam system on there. And we had a little flying squirrel demo. And the number one thing people complained about was they couldn't tickle the squirrel. And you just realize that oh no, you don't care about all of the optimization and getting all of the kind of like slam systems working with the visual inertial odometry getting all the mapping, which is like oh, cool, a flying squirrel. I want to tickle it and then you're like, Oh my God, or someone else was like, Oh, why isn't or a shadow on the ground. And then of course, my favorite one was, hey, can the AR cat move the cup? And then it was a real world cup, and you're just like, No like cheese. So I view number one thing, the tech is two steps away from a customer? And do you need to solve the middle step in order to get there. And that's kind of a difference for deep tech companies versus a view other companies were stereotyping a little bit here. But hey, you throw up a website, and you see, do people want to click on this website or not to gauge traction, rather than kind of building out the tech beforehand? So I think you need to figure out what is the middle step on there? And the second point is, how do you do the customer discovery process to validate that the tech that you're building actually solves that middle step. And you're not just thinking about it from a hammer looking for a nail, you're like, Okay, this is a very important technology, but like it uniquely solves this problem. And just being more on the problem centric and saying, okay, for Escher reality, what do game studios care about, they care about reducing their customer acquisition cost, they care about making cool stuff that people will like, click on and engage with or download. And they do like chasing after what is the latest and greatest kind of technology that's out there. But there's also the trade off of many of the ones who like to be on the bleeding edge might not be the ones to make the billion dollar games off of there. So then there's like a whole huge pool of indie developers who are like, the best and smartest hackers in the world, and I love them to death. Or from a building of a company off of there, there's kind of a bifurcation of small, small studios and ones who have made it. And occasionally one pops very quickly to go into the next bucket. So yeah, figuring out the customer discovery process to solve that middle step. And the third one, I would say is, we underestimated the power of messaging and marketing. So then simplifying your message down to be as clear and simplistic as possible for your end customers and your end audience. So that's both from a venture capital perspective, but then also from a sales perspective. So how do you go in there with like, the two bullet points, like, you look at a lot of marketing stuff for big companies, like you had Apple in the past, like think different, and it's kind of these vague values driven stuff, that's great when you've made it. But when you're kind of an early stage startup, and you go into, Hey, your marketing pitch could be one metric, you're like, we reduce customer acquisition costs gaming, by 40%. By offering a new path on there, you just felt like people who really care about that will like it, and the people who don't go away from it. So just that very clear messaging, like, you have no idea how long it took to come up with the phrase of you need to understand reality, in order to augment it, we did the reality understanding part. This is more from a venture capital pitch perspective, it was like they heard Apple launched AR kit, they knew there was stuff in the space, okay, they didn't know what was kind of the what was actually moving at the time, there's a lot of uncertainty. And they're like, Oh, if you do, you would need to understand reality. So then guiding them into that thought process of, oh, they're doing the reality understanding, then you can start talking through like data network effects of like, hey, if I map this room, anyone can use it. And the most up to date map is the most valuable. So then, as you can guide people on that track. But if you look at our initial pitches and pitch decks that we tried to do, it was, Oh, we're offering this and this was very vague. So just being very concrete. Those are

really fantastic bits of encapsulated insight. And I'd like to come back to this maybe this topic of advice for deep tech founders a little bit later in our conversation. But let's go back to Niantic for a second. From all of the other startups it is one of the big whales out there. But compared to the Googles and betters of the world, Niantic is itself quite tiny in comparison, but you joined the company, as you noted, the team saw your vision as compelling and worthy of being followed, they brought you on and your team on, you got to continue that vision unit building out helping to build out that light ship AR platform. And the pursuit of mobile AR which was the initial focus, that pursuit ended up expanding to include some headworn stuff, there was an appreciation I presume that at some point, the vision is that it goes from the phone to a pair of glasses, maybe in combination with the phone or maybe just the phone but somehow or desserts, the glass Excuse me, but somehow the glasses are involved for some sort of headset, maybe gonna walk us through kind of the the evolution of the vision and the strategy at night and tickets related to AR how it fit with the bigger picture.

So like I'm no longer at Niantic and don't speak for it and some AI stuff may be a bit dated, and I will be a little bit more careful in terms of what I say because I haven't gone through their PR team at all, but I want to be very respectful of them since they were very good to me. So when we started at Niantic, it was very much gaming and was like okay, how do we Build up the AR side of it, we built the AR platform. There was a flurry of acquisitions to go in there in terms of, okay, how do we make a an augmented reality SDK that will help, or Lightship that will help as many developers as possible? Because Niantic tons and tons of lessons learned of being at the forefront of making AR tech and thinking about how does gaming impact the real world? So then there's that, how do we help developers with that developer toolkit, then there is kind of the the visual positioning system which I know one of your past speakers Meriam, very dear friend of mine, co conspirator in some cases, then there on the VPS side, like that is again, turning Earth into an AR game board, if you want to have persistence, multiplayer experiences, having everyone synchronized and to one state of the world. And that was all done on mobile. But then, many people know there's literally billions of dollars and at the time, still billions of dollars being invested in space, hey, headsets are going to take off and they're going to have their own platforms that are going to be at least a periphery to the phone, or at most a replacement to the phone. We can discuss later in terms of what is the right path on there, because that was the ecosystem at the time. So it's 2019 2020, early 2020, before pandemic, where we were thinking about okay, Niantic AR SDK a or mapping company, or AR cloud company, credit to mountain already for that one. Then how do you make sure that okay, the mobile we that Niantic has transferred to headsets. So then what's public like Samsung made an investment in Niantic, and we announced a partnership with them or with Qualcomm. And there's other kind of different people because everyone would come to Niantic because obviously they want applications and the hand tick is at the forefront of AR applications on there. But then, you can imagine in some of those discussions that if Niantic wants to be more of a platform company, have more of the developer ecosystem and the data side and using the content to get there. Other people might just want the content side and they want the data side as well. So there's kind of interesting tug and pull in that that dimension should I want to get into too much detail on but then from there you can look at okay, fundamentally if my antics user base and and I cannot this probably is one of the least gaming people inside of Niantic. So like, Don't misrepresent Niantic is not thinking of the joy and pleasure and all the gaming stuff there's so many more people in there's so much more important. But from a pure AR perspective, there's a great user base that can be that has already been collaborating to build the map of the world that they play on, like a lot of the pokey stops are hurt crowd sourced. So then how do you make the transition from that to headsets in a way that keeps my antics kind of vision and goals aligned in there. Now towards that, if you think about people are building generic AR platforms through place mobile like hey, only thing it'll justify the amount of spin there is replacing the iPhone, so much money flying around in there. But fundamentally, tech rolls out because of specific applications. People forget that the iPhone was a phone first. And they forget that it started with an iPod. So if you look at any of my LinkedIn, they usually mentioned this every kind of month or two and like, Hey, start with the iPod first. So if you have a lot of the user base, and a product that people are rabid about, and they love, and it makes like Forbes news, whenever like you make a small change based off of the size of a pokey stop isn't 40 meters or 20 meters. And then people are an uproar when they post COVID. And they go back down to 20 meters what it was before. That's like national news. So like that is a very hardcore, amazing user base of people that are so excited about the game and the community that they're a part of. But then if we have that amazing user base have dedicated people, okay, what would be the right device just for them? So forget all the other applications just get down to how do you cast a Harry Potter spell? How do you throw a Pokeball? How do you make kind of an outdoor game console that can please a user base. And this is all at the time. And sadly still now, if you sold to one or 2% of Niantic user base, that would be the number one error headset in the market. So then if you just think about it from a stepping stone, and you don't think about oh, how do you get to 100 million devices? But how do you get to 1 million? How do you have like the number one here had set the best kind of map of the world for people to play games on or more and you have the number one content on there. That is a very strategically interesting position for a very, very unique company that there's really like there's really no other comparable to Niantic in the market, like very much stands alone in there. So it's just kind of trying to leverage the unique assets have to have even more unique assets and very competitive space in there. So that gets into like reasoning why a software company would even think about doing the hardware. And that's not saying that Niantic has the goal and been very public about, like, trying to manufacture it not trying to even like fully branded, it's more of a, how do we do all the r&d? solve everything for what is best for, say, for example, throwing a Pokeball, very simplified, very defined, what is the funniest experience? Like what is the capture mechanic for capture base game, and then you can think about, once you solve for that, what are a couple other things you can do with that technology. And that allows you to go into different directions, then I need to have a 50 grandparent classes that everyone will wear hold the time, we're thinking of a little bit more session based thinking of specific applications. And if you think about it, like from a game console perspective, people have like their PlayStation five, their X Box, their Nintendo Switch, and they come into they think through that, okay, I'm playing a game for the next hour, two hours. Rather than I need a little bit of heart rate information as I go on a jog, or tell the time or get some notifications or have a Google Maps app. As I walk around in the store. Those are very different applications. And what we saw on the market was there's kind of three different directions seems to be going and I can talk about that more now. Because it's kind of fulfilled what we initially predicted, there was kind of the virtual reality mixed reality kind of camera pass through, we can kind of call it AR, if you squint at it felt like really, it's a for many of your audience, it's a little bit of an eye roll to call that fully augmented reality. And then there's the smart glasses like Google Glass, there's other enterprise versions of that. Ray Ban stories, snap spectacles, there's kind of a how do you have like a heads up display, or at least the earlier versions of SNAP spectacles, maybe it's camera only, maybe it's a small heads up display, but you're just focusing on what is the bare minimum to get the form factor. And on the other avenue is you have immersive headsets that have like you're thinking about the HoloLens, and magic leaps. And there you have a full system, that's kind of everything you'd want out of AR, but you look like a stormtrooper when you're wearing it. So like it's not, it's kind of too big for what you're going after. And viewed that, from a consumers perspective, people weren't really going after immersive augmented reality. But if you're thinking about gaming, like immersive is one of the first thoughts that you have, which is why you see, like, Oculus has sold a lot, their MCU numbers aren't quite what they would want, but they're still in the hundreds of 1000s. if not millions, maybe, I don't know the latest numbers on there. But like their immersion is like the entire differentiator that you have on there. And then if you're thinking about it from an AR context, okay, if everyone's going for smart glasses for the form factor, which I'm one of the first people that would love 50 gram glasses that could do everything that a HoloLens or a Magic Leap to could do, but physics and refer to like Carl Gu tags session with you beforehand, and five hours later, you'll come into Hey, physics, it's a pain in the butt. If only only VC money could get get rid of that small problem. But yeah, if you don't have the immersive angle and view that consumer was not really being tackled in that way, then there's potential position in the market that you could look at to do that. Now that's to say, an idea of what does exist in the market. There's other games you can do with smart classes. And I believe Niantic has talked publicly about some things, or at least interest in there. I don't know anything more than you do. But like there's different types of experiences you can have on different form factors. But if you want to go for I want to see pokimane. Imagine everything in the real world have an amazing ingress style spy game, or a Pikmin. Seeing flowers in my front yard and other games built off of that, like immersive does come to mind. But how do you do that for a gaming use case, just limit it to the scope on there.

And so here Niantic is really doing all of the heavy, hard work on the r&d side, from the user experience, translating that to the hardware components and what's available in the market today, based on the physics and the engineering that's been done so far. And the gift that has been given here is is this kind of bundled together in a reference design for somebody to take up and help deliver this experience into the night into customers. So it's an opportunity and it's a device that kind of sits as you noted, amongst those other three major categories of smartphones, fully immersive AR and the video pass through VR stuff that we are going to refer to occasionally as Mr. AR. There was an article not even article there was a thread on Reddit that referenced a patent that was public. They really I think highlights some of the creative thinking that was going on and I antic about other form factors that could be envisioned for how to deliver a gaming experience. And it was in the form of a hat and a hat in the world. Pokeyman as part of the character as part of the character experience in some ways, and so it kind of it's an interesting embodiment of the technology. What was kind of the the thinking you were you and Marian were both listed on the patent as, as CO creators, along with Michael Miller. What was kind of the thinking there, as you were exploring these other ideas,

let's just say it's always interesting that Reddit can dig up from the bowels of the patent office. You can imagine having a hole installed device and if you move things around cut features, you can shuffle things around into different form factors that hat was, as you said, was on Reddit, like that is one of the form factors that have you expresses the creativity in terms of what you were thinking, like, if your goal is, what do you what is the best device to throw a Pokeball for Pokeyman? Go. And that wasn't entirely what we limit ourselves to, but it gives an idea for your audience in terms of the specificity that we are going after. Now, do I imagine that the next Apple device that they come out, it will be like a ash catcher in baseball hat? No, I would be absolutely shocked in that some of the people I know have kind of explore different areas around in there. But everyone really likes the idea of a small pair of classes. And then they'd say, Well, I've big enough checkbook that I'll go and fight the good fight against physics. And to be clear like that That wall is moving, or there's creative paths that you can get through that in there. But for Niantic, he said when started off is not one of the giants cannot spend $10 billion a year or clip things down and only spend $6 billion a year oh my goodness, all the cost cutting. So then you need to be creative about what are the different ways you can go after it. And imagine like a sorting hat from a Harry Potter style system and Ash Ketchum had, if you're thinking about Tron like goggles, if you think about other systems on there, like there's a lot of creative form factors that can fit the specific use cases. And as I think Mariam spoke on podcast with you, like, we tried out a lot of different form factors. Some of them we rejected immediately. Other ones were like, hey, that's interesting. Some of them we filed patents on, because we thought that they were promising. And then I don't know where things are right now. It's been outside of Niantic for about a year now. So a lot can happen in a year in terms of job and time and also priorities. But towards that, like I, I think that thinking, what is the right hardware for the right application, in a form factor that consumers can enjoy it? I think there's a lot of interesting kind of questions that not just looking at tech, but other companies can deal. Because if you say you want full HoloLens, two style features, and like 30 grand pair of classes, you're not going to get there this decade, maybe not even next decade. But if you simplify down, what is the application you're going after? That's fundamentally one of the things I believe in, is like, what are you actually trying to solve? And any AR person listening here? Shouldn't have, we want to do a general purpose platform? How do you actually build for what are the applications and for gaming, that's kind of one application. And you just need to start somewhere, again, start with the iPod, that gets enough traction to get to the iPhone, which again, starts as a phone, and people were using blackberries at the time. And that explodes into the world's most valuable company today in a hardware ecosystem where everything fits together. So it's what is the first domino, I think, should be the number one stuff that people go after. And the Reddit article highlights an interesting path.

Fair enough. So these these kind of major buckets that you've identified, you just kind of brought this great point forward again, which is really understand the problem you're trying to solve. And the device should help you solve that. Because building an everything device in the perfect form factor just isn't possible today. So you have to be selective, you have to make some trade offs. And each of these kind of major bugs that you identified with the rest of the market is pursuing have made consciously or subconsciously some of these these trade offs between the kind of the video pass through, which was the apple vision pro that's coming out, you have the the fully immersive AR glasses, the HoloLens and the Magic Leap to the world. And you have the smart glasses. As you noted, the spectacles of Ray Ban stories, and Google Glasses are kind of most most famous of those. You should kind of think through those four factors and, and the implicit or explicit trade offs that they've made. Where is the longer term path I want to kind of come at this in a couple different ways. But the first is, if you had to bet on only one of those three buckets, where do you place your bet?

We can get until later in my current company in terms of placing a bet on there, but the thing is, is it depends on what the goals of the company going after it like for Niantic, which is very outdoor social, like hey, as you're walking around in there, you It might not make sense to go after a Oculus quest style virtual reality device and add some cameras on there. Because I even think with the Apple vision Pro and all the great designers work they've done. If I see someone like walking down the street or playing with their kid in the park with that on, I will be absolutely shocked and like be like, Okay, that's a that's a do. I think there might be business people that are on kind of 10 hour or Trans Pacific flights that will be wearing that. Yeah, I can see that. But for, like different companies and what their goals are like, do I think snap? Again, no insight, knowledge or anything else? But like, do I think that they will be going after virtuality? They haven't announced anything on they don't see it. But then if you look at meta, obviously, they said, This is something that makes sense for us. And they have enough resources that they can have many different paths, going at once to which is both a pro and a con. From some of my friends inside there. The Curse of plenty, I think it's called. So from a betting on what are you going after? Again, it gets down to what is the first applications you're like, if you want to replace like a game console in your house, if you want to replace computer monitors, like they're taking a virtual reality and doing a camera pass through for other applications on there. That makes a lot of sense, if you're trying to go after, like the phone ecosystem and having smart classes on there. Like consumers already buy expensive pairs of classes. And that's one of the things I fundamentally believe in is don't go after a new market. Let's get into our future conversation on like deep tech, but like, what are people already spending money on? And how can you make that better, so they spend a bit more money. So like people already buy game consoles in their house, therefore, like going with an Oculus quest, starting there, I know they've dreams beyond gaming, but most of the attraction is in the gaming space. If you think about the Apple vision Pro, like I personally have over $3,000 of computer monitors to make up a large visual viewpoint on there. And I know Carl goo tog might disagree with a lot of his posts on why mixed reality is not actually good for computer monitors. But people are spending money on that I can see a trade off of you're giving up ergonomics you're giving up on maybe some readability and resolution. But you can have a very, very portable 100 inch monitor that you can take wherever you want. So you could have that in a coffee shop, you can have it on the plane, you could have that in a hotel room, it's all very personalized on there. So it can be better than a laptop. So there there are trade offs on there. But it depends what you're going after. So I think that if you want to go for mass consumer adoption, you probably do need to go with classes, and go for that like sub 50 gram or at least from some of the numbers that I've looked at probably sub 75 grams about the limit. But since I have these big companies, they have their own trade offs of what they view would be acceptable, I think, I think volume is a little bit more of an issue in the higher of like plus 50 to 75 grams. But on there, like if you want to do that everyday where you're probably getting into classes, because what do people normally wear, like I will turn back my car to go and pick up my sunglasses, if I'm going long drive, because they don't want to be squinting, you don't put the visor down. That's something that I would do a u turn on within like a few blocks my house, I need my glasses, people with prescription glasses, okay, those are hundreds of dollars if you could have them being smart. So those are already existing consumer behaviors that and I view most people, I don't know the exact numbers, but like 90 plus percent of people have worn some classes in the past year. And many people do so on a regular basis. So if you can fit into the rave and story style thing, providing a little bit more utility than just camera taking, I think then there's devices that can go in there that already fit into existing consumer behavior. Now, would I want that to be fully immersive? Great. Yes, I would. But you need to make a lot of trade offs to get in there. And then you get into okay is the extra cost and form factor providing enough value. My view like the Ray Ban stories with the two cameras, taking first person viewpoint videos, and like that being positioned a little bit more towards the influencers, like I can see that being valuable to certain degree, but it's more niche. The question is how do you expand beyond that niche? And I still do like just having the metric of how do you hit a million units sold? to then get to be the number one AR headset? Because actually this could be a good question for you. Do you know what is the number one selling AR headset? Not mixed reality are of all times of all time or not? I don't know for military so let's get military maybe there's I'm

gonna guess it was some super specialized, simplistic gaming experience tied to some major IP, like a Star Wars, something

yet the Disney Lenovo Star Wars Jedi challenge where their pitch was, hey, 200 bucks. Do you want to swing a lightsaber? Darth Vader in your room. And that's sold almost a million units. I don't know the exact number, but it was like, over 600k less than a million. So if you think about that, compared to, I think there was an article like a week or two ago, about Ray Ban stories that sold 300,000 units was like five years after the Disney Lenovo Star Wars Jedi challenge. And like the guy who heard about that project, good friend of mine, really like equities is kind of pragmatic thing. But think about, hey, if you had another product that, okay, you have it at $400, it has a little bit more games on it or other experiences on there, could you do better than that kind of watermark for sales. Now, they didn't have the usage on there. But usage is something that plagues like a lot of the virtual reality mixed reality and smart glasses. But like, still, they sold like almost a million units. That is amazing. And that doesn't get celebrated enough in the market? And it's a question of, okay, how do you sell that many and then make it sticky? And then how do you do a gen two of that? How do you get to a Gen three? So like thinking about it from that more ground up perspective, rather than let's make a new platform for everything? I kind of took your question took it off in a random direction. But let me know if you want me to jump back.

So along those lines, you're getting at this notion that traction is key. And for traction, to really take hold, in order to deliver those sorts of unit volumes, there has to be some sort of use case some sort of application that is driving that behavior, that sort of traction. So you talked about this, this idea that there are certain behaviors, certain devices we already have in our lives, that are so important to us that we will go out of our way to make sure that we have them with us. And we don't forget them. You notice sunglasses or prescription glasses are some examples. But how do you articulate this notion of the applications that will gain actually have the potential to gain of traction to justify these different hardware verticals? And then kind of what makes them viable?

Have you fundamentally what makes a viable? That's such a hard question to answer like that something needs to ask in the market. So if you like even on the early days of Google Glass, like a dozen years ago, you had so many VCs saying, Oh, this is great. And like this is going to change things. And the book The hard thing about hard things like the last area, they applaud Google for the amazing success, Google Glass will be and now we're a dozen years later, and glasses died twice. And I predict will probably die a third time before something takes off. Sorry, to anyone on the Google class team, I realize that I respect everything that you guys have done. But at least we're making it viable. Again, it gets down to focusing on what two people buy today, what our existing behaviors today, and how do you make them better, like a great application that I've always loved. And I almost pulled the trigger on like even doing a start up on that is like a Kindle e reader. Like the marketing is already there, get bring your Kindle reading classes. And then it just focuses on like showing books and people already buy Kindles. They're like 100 100 bucks with ads, like tuner bucks without ads, you're gonna get the color ones, I think you're getting up to 350 bucks. So you had a $500 pair of reading glasses, that you just focused on making that small enough. And then there you have, it can be a heads up display, you don't need any insight of tracking, you don't need a camera. Like you can buffer all the content in there, you can have the cheapest processor possible. You don't even need a microphone, the only thing that you need that's kind of special is like how do you turn the page? Because you can load up the book in the app through Bluetooth, offload it, like but that's something Okay, how many Kindles have been sold in there? How much money does the Kindle bookstore make for Amazon existing business like, and then if you talk to Hey, so you have a green only display of some plastic, very, very simple small field of view or like 3035 degree field of view optics, that's about the size of a book at arm's length. And your value proposition is okay, you don't need to prop up your arm like is all of us do is we're trying to read a book, you're trying to hold it in there and you're like, Okay, your arm gets tired, you switch over to one side, as you go to the other side. That's an idea that I think is kind of good from an application standpoint, could be viable. There is I think one or two companies exploring it. I know one of them is doing it more as a mixed reality system, which I think is a little bit of a mistake, like seeing the real world is important. But like there's different trade offs you can do in there. And there's some interesting problems and reason why I didn't go after that as a startup is okay, it hardware doesn't make the money but it's all in the bookstore and you can't compete with Kindle on the bookstore. Or it's very, very hard to compete with Kindle on bookstore because once you show some traction, they'll just spend $10 million your all your supply chains have now moved over to Amazon and then you're done. But like just focusing on what are very simple, small applications. Again, getting back to what is the iPod, and if you think about before the iPod like there were some bulky mp3 players out there. I still had my like CD player that if you twisted it as was planning due to giant gyroscopic it would skip it And there, then you had like the Sony Walkman people already buying music playing devices. And there's hardware advancement. I think it was Toshiba that just made like the world's smallest hard drive. And then they just said, Okay, how do we package this up in a way, very quickly that works for consumers. And then their first year, I think they sold 600,000 units. Their next year, they sold 3 million, then I'd like started to grow up like into the millions from there. But that was something people already spent money on new tech advance that completely unrelated to music, but from a systems engineering perspective, like the team at Apple kind of connected the dots on there. But if you didn't have the iPod, do you think we would have had the iPhone? And if Toshiba didn't just build cool tech and saying, Hey, we want the world's smallest hard drive? Would that have unlocked it at that time? Like there's a lot of stuff that you just need to kind of find the opportunities find that's viable, but fundamentally, it's just getting down to what to if you're going for a consumer device, enterprise is a little bit different ballgame. But even there, you can fit into p&l and politics, rather than, like what are consumer spending today. But I think game consoles, people spend money on that computer monitors, people spend money on that sunglasses, people spend money on that Kindles they spend money on that. So like, finding what is that niche with an already existing budget? And then having a clear reason why is it better

is you have now kind of processed all of these opportunities that kind of have the characteristics that you describe, what is the one that you are excited about? Well,

I can say that, but I'm dedicating my self and my acquisition proceeds towards my company, Okamoto. So for that, it all kind of boiled down to a problem that I actually had after my first kid. So I guess circling back to the beginning, if you remember the baby formula shortage last spring, last summer that made like national news, and then you want to talk about personal pain point, feeding your kid, it's pretty high up there. And then when you go into a whole bunch of stores, and they're all out of stock, then that was a major problem that I faced. So and then in digging into that ended up on finding a very interesting problem. And I swear, it wasn't me with a hammer looking for a nail even though once you can figure it out the whole market then looking like oh, that's a very Ross finman problem.

The story continues in the next episode, I think you'll really enjoy the rest of the conversation, and thanks for listening