The AR Show: Japjit Tulsi (Matterport) on 3D Digital Twins and Building Strong Engineering Teams

4:35PM May 3, 2021

Speakers:

Jason McDowall

Japjit Tulsi

Keywords:

build

space

capabilities

people

capture

twin

company

digital

understanding

understand

camera

team

platform

happening

case

knowledge graph

information

object

scan

data

Welcome to the AR show right dive deep into augmented reality with a focus on the technology, the use cases and the people behind them. I'm your host Jason McDowall. today's conversation is with Japjit Tulsi is the CTO at matterport, a company building a set of technologies to accelerate and simplify the creation of immersive 3d Digital twins for real estate project planning, hospitality, insurance and more. Prior to joining matterport, Japjit gained more than 20 years of technical leadership experience, including as CTO of Carta, a tool for investors, founders and employees to manage their equity. He was also VP of Engineering at eBay, leading engineering for new product development, including their AI powered shopping assistant. Earlier in his career, he helped to build products at Google, including Google Analytics and YouTube's innovative content platform. He also held numerous senior leadership positions at Microsoft, along with entrepreneurial pursuits at StumbleUpon. Japjit also serves as a board member at grassroots ecology and environmental education and action nonprofit. In this conversation, we get into the technology at matterport. And how Japjit and his team are leveraging machine learning to deliver greater insights with less data, essentially enabling you to teleport into a space, but their vision is to do more than capture physical spaces.

One of the things that we have always said from day one, capture is just the beginning of the story. So capturing these millions of spaces, is just getting us to the very initial phase of the company. The long term plan is really around insights. So the more we capture the more we understand real space, the more we can provide you insights

Matterport's Cortex AI and deep learning algorithms leverage the data to bring a unique index of millions of spaces and new insights into the operational efficiency of buildings around the world. We talk about the company the technology and the roadmap in Japjit shares some great advice for building and maintaining strong engineering teams. As a reminder, you can find the show notes for this and other episodes at our website, the AR show calm. Let's dive in.

Japjit, you are an avid pilot in your spare time, what was one of your most memorable flying experiences?

I've had a few to be perfectly honest. Every time I go somewhere now I actually like to go fly in location. And so one of my fun moments was I was in Geneva and went out with a veteran pilots believe he was World War Two and just flew around the lake. And what's cool about Geneva is like Switzerland's right there it leaves right there, France is right there. So you have to go skirt all the boundaries. As a result. I did something similar in Slovenia, and flew to their tallest mountain which is 13,000 feet and down to like a beach town and landed at porta rose. So it was a beautiful diesel engine flights that was kind of fun. flown a helicopter where we had the that was me but helicopters can auto rotate. So which means without power, they can land. And so this Vietnam pilots level pilots, basically auto rotated down into a ravine landed on the on the banks, which was fantastic. So I you know, I love flying. And this is definitely I've had a few great experiences.

It's amazing. What was the original impetus to begin to learn to fly.

I've always wanted to fly since I was a child, to be perfectly honest, they actually even tried for the when I was in India, where I grew up, tried for the Indian Air Force, but my eyesight was in great danger require like just perfect vision. And so I couldn't get in. So I've always wanted to fly as when I moved to the US and to the barrier. I said I'm going to learn how to fly and I did. So it's great.

Amazing. That's so amazing. An extra level of freedom and adventure possible. I guess with that

that person is actually go back and add one other story to your collection. I had the pleasure of going and flying a fighter jet. It's like a trainer fighter jet. So it's the L 39. It's not like the fastest or anything but, you know, does lots of acrobatics. And if you see them in a show, they're usually there and went and did a bunch of acrobatic lights through the barrier, which is a lot of fun. So

that's really cool. I was in the US Air Force. And when I was in college, they did some Bank of aptitude tests. And one of them had to do with your ability to kind of perceive your position relative to some other object or spacecrafts position in space, some sort of spatial understanding test, and some of the tests outside of it. They determined that I was well suited to be in a fighter jet. But like you My eyes were not good enough to be a pilot at that time because at that time Air Force did not allow surgically corrected eyes. There was concern that the The weakness in the the walls of the eyeball would cause problems at high pressure if you're suddenly to depressurize the airplane and terrible things might happen. But I think since they've, they've updated the perspective on it. Anyway, one of the summers that I was in college, I got to spend three weeks maybe with a fighter squadron in England, flying f 15. Eagles in this experience was amazing, a lot of ways. But we got to spend some time on the simulator. And we got to go up in an airplane. And the F 15 Eagle this particular model was a two seat version. And so that was the the real pilot was in the front seat. And then I as the young cadet was in the backseat, but for a decent chunk of the flight, I had the stick. And it was truly extraordinary experience. It was one of those where you have to you have to wear the suit where they pressurize the suit, right, where they got their squeezing basically your calves, so the blood doesn't all rush to your toes and out of your brain and you pass out you have tried to keep the blood, the top part of your body as you're pulling these really intense G's through some of these aerobatic maneuvers. And it was, it was a truly glorious and wonderful experience. But oddly, after that experience, I decided to not want to be in a fighter jet. And it was because I wasn't allowed to fly is because my primary responsibility would have been in the backseat of the jet responsible for being aware of what's happening around and deploying payloads, or, you know, doing the operational mission behind the driver in the front seat. And after having spent three weeks with a squadron of fighter pilots, I realized that I have a personality conflict with the standard stereotypical Air Force pilot, at least at that time, given who I was then. And I just, I had a hard time imagining the commitment that was required to ultimately be in the backseat. But I guess for me that the joy of flying was was having the stick and not just being being a passenger. So I can appreciate that.

I totally agree. We have one of those t shirts that say like in life, you're either a pilot or a passenger. So you obviously wanted to be the pilot. wanted to be the pilot?

Yeah. Let's jump back here. Yeah. So in early 2020, so I guess it was about a year and change ago, you joined matterport as the CTO, what attracted you to the company?

Yeah, I would tell you like a lot of the last few companies that I've worked at, it ends up being first and foremost that people the the people that I've worked with the people that I want to work with. So that's usually number one. Number two is the technology. So for about the last decade or so I spent a increasing amount of time, you know, thinking about machine learning, deep learning, and really thinking about what are the next stage next generation of things that we need to build is that in the search engine Lan, is that a knowledge graph that needs to be built out is that computer vision is that multimodal assistance. And so I've actually, you know, spend time over the, over the last decade building out a variety of these different types of systems, including, you know, recommendation engines, and such. A motherboard came along there was like, you know, two of my my core reasons to join a company, the people, there's a few folks here that I've worked with for many years, RJ who's the CEO, I've worked with Google, as well as eBay. And similarly, a couple other folks from the organization that I've worked with, over the years are at matterport. So you know, that makes it a instant attraction. But then also, I've actually worked with RJ at eBay, I would tell you like it was one of the best leadership teams I've worked out. So I knew that he would have collected a great bunch of people here as well. So that was part of the impetus. And then second, the technology, like motherboard is a deep spatial indexing company, we scan we understand what is as part of the scan, a building a space, a factory, Starbucks, Home Depot, whatever the case may be, we pull that all together, we understand what's happening in that space. And then we provide insights from there. We're also obviously well known for our virtual tours, residential real estate, virtual tours as well where we provide a lot of that capability in space. Maybe the last but not least, is my airport provides like a hardware and a software component which I've never really honestly done hardware. I've always been interested in doing it. I've always like dabbled and built out a slideshow of different things, you know, on the side, but never really like production where the hardware equipment and so that was like the third thing that I would rather like also attracted me to with modern

infra matterport case what is the hardware is a specialized cameras for the capture of the scene.

We do we produce a camera or two. So we've had matterport Pro one, which was the original camera that the founders built in house developed and manufactured actually all here in the Sunnyvale facility. And since then, they switched over to the pro two, which is also a built in house manufacturers in the same facility. It is a specialized camera has active depth sensing, built as part of the RGB capabilities along with the RGB capabilities, I should say, over the last year and a half, though, we've also expanded the scope. So really made it more of a platform. So we welcome other cameras into the ecosystem as well. So that includes 360, cameras, other depth sensing cameras, and as recently as May last year, we launched iPhone, as well as a active camera that you can incorporate, including iPhone and iPads with LIDAR, should we were happy to take any and all camera types that we do build our own series custom cameras? Well,

yeah, you'd mentioned that real estate was one of the areas that Metapod has found a lot of traction with, I've seen a few as I've, you know, paying attention to my own local community occasionally run into a matterport experience in one of the houses in the local community that's for sale, as always a really nice experience of being there and having a better perspective of, of what's going on. But can you walk us through maybe that use case for a typical realtor for a new house listing, and some of the other ones that kind of go above and beyond and take more advantage of the platform, that metaphor is building?

Absolutely. And obviously, the metaphor, residential real estate is something that's publicly available as well. So most people will, will know us from that, because that's what's publicly available. And so in a typical case, when you have a real estate listing, that that occurs, most time the realtor, you know, invites a photography professional to kind of come in and take pictures of the location. And a lot of the photography professionals over time have also incorporated, taking him out of Port camera and building out a, you know, digital twin out of that space. And then providing that using our capabilities, our platform capabilities back as a as a showcase for the end user. So the typical experience is the professional comes in, they, you know, come in, if the house has been staged or not, they can actually come in and look at place the camera. So every time you actually walk through the space, we'll see a few pucks that are lined up box out these little round discs on the floor. And that's where the camera had been placed when the original photography was being taken, along with the optic active depth sensing. Once you've done that work, they uploaded to the cloud using our app. And we process and then make that available back to the professional to be able to push on to the website. So it's a relatively quick affair, like folks can come in and you know, image or create the digital twin off this space. In relatively no time these guys are professionals, they know exactly what to do, and then go from there. The cool case I would tell you is there's a few actually. So we've really built out the platform for matterport. So let's say your space was completely empty, you can go to one of our partners, and they will completely virtually staged that space for you. And I'm happy to send you some examples, if you like. And the way that we've captured this space, we have some really amazing capabilities. So my favorite example is we had this penthouse in New York, that they captured that was completely empty, that was then staged by our partner. And using the platform capabilities, API's and integrations that are built in into my report. And you can see like where to place the mirror, if you actually went and looked at that mirror, it would show you the skyline in the in the backdrop because we know exactly where the dimensional capability, what you should be seeing through a window. And we can reflect that back into the mirror. So that's where it becomes really cool and interesting from both a platform capability perspective and what the digital twin offers for you. And then similarly, if you're a more to yourself person, then you know today, you can actually download the matterport Capture App. And just using an iPhone and very soon an Android phone, you will actually be able to take and create a digital twin, just using your phone. So you don't actually need any other specialized camera capability, just the phone on on your iPhone, and soon to be Android, you'll actually be able to create a full on digital twin.

You kind of describe a couple of examples that one you could almost do with the regular 360 camera to some extent, right, the real estate sort of experience where it's really about just understanding what is already there, as it's already been physically staged.

That's right, and you can so you can actually use 100% of a 360 camera and do that you can use an iPhone camera and do that Or you can use a motherboard branded camera and do

you begin to take it well beyond the 360 camera because not only are you capturing the photons that are coming back, we actually have a sense of the the environment itself, a very detailed sense of that environment to describe the second scenario, whereas if you captured empty, and now you're creating all this, this digital staging, so it's all completely virtual staging, which could look different, depending on your target client, and how they could imagine the space, you could really customize that for them, theoretically, and for end users using LIDAR on their phones, they can create their own digital twin, what are the sorts of capabilities that take even more potential advantage of the fact that it is truly digital twins, that you have all of that spatial information of that environment and all the objects within that environment as well.

So we have a deep learning slash machine learning system that we call cortex. And one of the things that we do do, as a result of capturing all the spaces, so we have close to 5 million spaces, or digital twins in the market today, more than half of them were captured with the matterport Pro two camera, and we definitely have a lot of active depth data available to that. So one of the things that we did with cortex was we trained a neural network that allowed us to take all that information, and build a model that lets us very accurately predict depth. So which is what to your point, we are now able to take in 360 camera imagery, we're able to take in, you know, a DSLR, 360 panoramic shot, or a iPhone, you know, panto as well, and build out active depth data mapped out and then give you a a mesh format, that is almost as good as if you had taken it with an active depth sensing camera. And that capability essentially makes it that we are able to offer that all of the other platform capabilities, around staging around measurements around our ability to go in and you know, let you integrate with virtual staging, or other types of platform partnerships that are part of our platform ecosystem. And make that a reality just based on whatever camera type that you have, or whatever camera type that you feel comfortable using for a particular job. Because it will also be job specific. Let's say for example, a factory where you want very precise information about, you know, your line manufacturing, maybe some aspects of where your wiring is coming in, or piping is going through, you want super precise information. So you might actually choose a different camera type with lots of depth data, maybe even active LIDAR depth data in there. If you're using you know, I just need a update on my existing model, where I'd forgotten to scan a space or so you can augment it by just going in with your with your iPhone, and then just adding a scanner to to that existing space and making that a iPhone scan. So this allows us to have flexibility on capture on one side. And then the cortex basically can take care of any differences in the style of capture that was taken with and then we can produce a output that is, you know, makes us the industry leader in the space.

Amazing. So you're able to based on all of the actual the higher quality depth captures, you're able to kind of figure out through the machine learning approach, using regular RGB, a good estimate for what the depth likely is in that RGB scene, that is correct and create a nearly equivalent, I guess, level of precision in that more generic RGB camera capture,

right. And now I'll never claim that we're like exactly precisely the same, but we're pretty darn close. And especially for spaces that have you know, large angles, we can get very, very close. When you get down to maybe the object level, maybe we're not as precise because we don't have enough depth data for your specific object level depth data. But for the space, we're pretty pretty darn close.

The company matterport has been actively pitching new investors recently, as we will talk a little bit more about later around the you know, the process of going public in in raising the funds through reverse mergers back process. As you were out there pitching and selling the story to these new investors, how do you describe the company's grand vision? Where are you going?

Great question. One of the things that we have always said from day one, capture is just the beginning of of the story. So capturing these millions of spaces is just getting us to the very initial phase of the company. The long term plan is really Really around insights. So the more we capture, the more we understand this space, the more we can provide you insights, a lot of our, what we call the dbo the design, build and operate, this could be a retail store, this could be a factory, this could be a system, that construction crew, they're all looking for ways of optimizing their footprint, and how, you know, they can actually use this digital twin in new and interesting ways. So I'll just give you an example over the last year, most of the world has globally and is a global problem has been going through COVID lot of use cases, where you were actually able to send in a person to go inspect something is no longer possible. So it's kind of our design, build and operate. Use Case, insurance is an example of that, insurance incidents are still happening, but an insurance adjuster can't actually make it to a particular location. So one of the things that we can do is send a hazmat certified capture technician to show up to the location, even during COVID. And this is global, even during COVID and come in and actually scan that space. And then insurance adjuster UK and in the safety of their house, be able to manage making that adjustment possible. In a similar vein, we've got a power company, they are required by law to actually go inspect all of their facilities on a regular basis, you know, in the old days, they would send out a crew of you know, three to five people that would go in with a checklist and you know, mark down all the bases. And in the new world, they are actually going to scan that digital twin on a, you know, daily, weekly, monthly basis, and then be able to go inspect that digital twin, because we provide provenance of when that twin was taken. And then you can actually inspect that in the exact same detail if you were there in person. So the operate space has become really interesting. Obviously, you know, residential real estate, commercial real estate, nobody can actually show up to some of these locations, universities, people can't go do tours of the university when they wanted to. So now they can actually go figure out where they should be going on a regular basis. So the list is endless in terms of capturing these spaces, and always has been. But now we can actually go in for the especially on the operate, use case and showcase, my stereotypical example is take a meeting, coffee chain, they have, you know, 35,000 stores around the world, they need to know, do they have all the social distancing guidelines followed. And there are different guidelines and the us at sixfields. In the US or in Europe, it's 1.5 meters, did the, you know, their social distancing, little stickers land at the right spot. And through spatial data inside, one of the things we can provide them is like, here are all the locations that have the right stickers here, all the locations, I don't have the right stickers, and not only we can, you know, give you the counts and and show you the Delta changes from what you designed, what was built and what's being operated. But also, we can actually show you this case is because we've actually scanned that space. And so they're also in the process of going in and saying we want to scan this on a daily, weekly basis. Because they're also changing merchandising, changing the layout of their stores, they're changing how the signage is pushed out on a regular basis. You know, one thing I didn't realize before I talked to them was, a lot of these stores are changing in a subtle way. So they go in and tweak a little bit, because they want you to feel fresh every time you kind of come in. So that design is changing in the corporate headquarters on a regular basis and then being passed on to individual stores. And they want to actually be able to manage and see how many of their designs are actually being followed and, and push through on a regular basis. So our ability to be able to visually determine what has changed. Where has it changed visa v the design of the of the location because that's all in computer software CAD as well, we can now actually start providing that insight so we can start providing insights like these many stores are out of sync with what your design was, here are the the actual changes that they didn't implement. And then you actually almost query that as a case with another example of a popular space sharing, you know, where you go touchdown desks and so on. They want to be able to say, when you book how many windows that are east facing that have a conference room of 16 people or more, and can you find those conference rooms for us in a particular city. So you can imagine that kind of spatial data query is something that only we can do. So that's really what is the next phase of, or what are the next phases of where we want to go. Second part is obviously platform where a huge platform company platform play. There's lots of integrations that are built in already into the system, whether whether that's with construction software, with architecture software, with insurance software, we will continue to expand that as a as a platform play as well, to really build out an ecosystem of partners and vendors, all the way from the capture side, to the usage side, because that's what people are doing today, on a much more manual basis, we're helping them automate and build a digital twin, and make that even faster and stronger. Amazing.

One of the analogies that immediately came to mind as you're describing that is that as the DVR allowed us to time shift our consumption of media and of course, it's completely transformed with streaming over the last handful of years. matterport is allowing us to space shift to be in a place without actually being in a place and to have the same level of fidelity. To experience that place. Or to understand that place that we would have if we were, we were actually there. And then to make that variable. On the back end of that this is really interesting, powerful.

That's exactly why we got to teleportation, just for giggles, on our side, because you can teleport into space and the equivalent of being there. But you can totally imagine that museum use case. And we've lots of partners today that actually do that, with museums, museums are hurting, I don't know, if you've looked at the data over the last year, they expect at least in the US 50% of the small museums will close shop, because they don't have enough foot traffic. And one of the advantages of actually having additional spaces, suddenly your market opens up to the rest of the world. Now anybody can show up to your museum. And so that's really the power of the digital twin that we we truly believe in. And we want to make sure that we can offer that as a as a capability to our clients.

Yeah, that's great. So you've talked about this cortex AI is the label, you're putting on the set of capabilities of analyzing that information, whether it's stitching together RGB data and inferring depth, or understanding the space well enough to be able to categorize it. These are windows and this is the direction all the rest, you'd kind of described, as you kind of think about all the things you need to do going forward and building out this platform is cortex AI, what's particularly challenging about building out this type of AI solution.

Word AI is always an interesting one. But let's, let's take it for a second. To me always the the problem with AI, is having enough context having enough knowledge about the system, do we truly understand the scene that is actually occurring? Like, can we infer other things? After we've understood that scene? Can we put the whole space in context? So, you know, can we put multiple rooms, if it's a residential house, and then understand the style of house that it is? and start to infer that? So to me, it's always about how do you get to that next level of understanding? because anybody can go in and say, This is a room? Or this is a factory? Or or this is a power plant? But can we actually put in more understanding all the way down? As you said to the object level? What are the objects in the room? You know, how, how are they labeled, understand that in different languages, because you know, we're a global company, across the world, different languages, different styles? How do you actually put all that together into a knowledge graph, that then you can push together and use it to understand even more and more so one of the things that, you know, I'm particularly proud of from my from my past, you know, having now done this for a few years, was when we were able to build a knowledge graph, that really understood, you know, down to the object level, and then also understood that in different languages, understood that in different contexts, and then provide that as a capability that then could power queries and so on, for the customer for the user. That is non trivial to do. So that's where, you know, this is why this is not not an easy task, and no one's done it before, is you can go in and understand a object you can define that here's, you know, five images of this particular object go find that for me, that's pretty easy. How do you actually then make that a contextual understanding of what does that object do? Well, how does it fit into the space? Is it in the right location? What if this object moved from the previous gown to the next scan? Those are much harder problems to solve because you don't want to just do plane Delta detection. You actually want to understand where is that object going? And how is that moved over time? And you know, in that prominent coffee chain example, if you move the signage if you move the layout, it's the same item, same object, it's just moved layout. And how do you understand that if you were doing inventory analysis and management, how do you know that the inventory has actually moved from x to y? And, and you can now count that as part of the same inventory, versus like, you know, if you moved it from oil to oil, so those kinds of understanding at the very core is non trivial to do so. To me, that's what makes it so interesting.

to kind of talk about this notion of the digital twin. In this sort of insight, you're gonna be able to derive from having captured this space? Well, how do you imagine that really connects to this broader notion within the augmented reality community of that vision of a digital twin? Are they the same thing or what we talking about in the broader AR community, the same thing as, as what you're creating, as a digital twin for all of these private spaces?

Yeah, private and public spaces. There's all kinds of interesting capabilities that our partners are already bringing on board. And you'll hear me say the word partner a lot. It's mostly because we think of ourselves as a platform company. So we are obviously going to do some things and enable a lot of things for the rest of our customers and partners to do. And that could be like first party, second party or third party partnerships that we have in place today. So specifically, as it relates to AR, there are some capabilities that folks are already building in, we've long had both AR and VR capabilities within the ecosystem, our ability to do cutaways build an idealized mesh of office space of any type, and then offer that back to the for the user base. For them to be able to either build a AR experience, or a VR experience is important. We've got catalog items that you can imagine that go into the virtual staging example, today, in the pure AR experience, you would end up using your phone and kind of placing an object, a object within a space. With our digital twin, you can place many objects. So if you go to our SDK site today, one of the examples that we show is your ability to, as I mentioned earlier, virtually stage space, you can actually go build custom experiences just using our SDK that allow you to place many, many objects in that space. The other example there that I think might be really interesting, but you can imagine working in the air world equally well as well, is placement of cameras. So if you wanted to look at a security camera, in a construction site, or in a factory or secure sites, you can imagine you want to know what the field of view of the camera is, what does it actually see, as a result, and having a digital twin actually gives you that capability of what would that sweep look like? What all would it miss? What what what all would it see. So our ability to you know, take that real space down to the dimensionally accurate viewpoint, and then be able to map that for any of those capabilities, is what I think makes us particularly unique. I don't know if I can tell you like here is the exact killer AR use case. But I know that once you offer that out to the developer ecosystem, people come up with some really interesting ideas and what they can and will do with the platform. So go check out our SDK, and some of our partners, they've done some amazing things with the asset.

Yeah, an amazing set of building blocks that you're creating there. That's right. Jumping back to this comment about matterport. And its evolution as a company, right, it's been publicly announced that you're going to become a public company through this special purpose acquisition company process, the spec process, which is been all the rage for the last handful of months, even though it's been around for for really decades, fundamentally, what comes from all of the money that's going to get raised? The point of going public is not simply to create liquidity. It's also a fundraising exercise, like many other fundraising rounds, but here, you're likely ending up with several 100 million dollars. Where do you invest that money? Where do you see the biggest growth opportunities for the company?

I think you'll you'll see that as part of our back deck as well. But really the space understanding deep spatial index, and then being able to query that index is where we believe is the next phase of our journey. We've done a great job capturing where you're at 5 million plus models, we call them spaces on the management, that's kind of our atomic unit. And really pushing that using that understanding and making that even more powerful. A lot of our enterprise clients are really that's what they want us to do is under understand in that space, I know this kind of sounds, you'll be like, really, that's the use case they want to do. But think about it, like if you are a large commercial space provider, and you've got 50 billion square meters of space under management. So most of these large scale enterprises, that's how they think they think about this, the square meters of space that they are running, and you need to know how many sprinklers and this is why I said, like, it might even sound boring, it's like, how many sprinkler heads are there, I have to go and inspect those on a regular basis, I have to go change them on a regular basis, how many square meters of carpets Do I have to go get? How many windows Do I have to replace, and today that literally have to do that manually. So how do they get that to a way of making that additional twin, and then rescanning, that space, you know, every five years, three years, one year, one month, so they're understanding the changes, if you're a construction site, a large scale construction site, and you want to make sure that everything is actually getting documented on an accurate basis down to the dimensional detail, using our capability and documenting that is superbly important. So there's so much both in terms of, you know, capturing, documenting, and then understanding and then utilizing, that is there, we've got, I think the math was there's $230 trillion worth of real estate. In this world today. We've got everybody in the world, if there's what 7 billion people in the world, there's about a billion buildings or apartments or, or spaces, just at the consumer level that people want to be able to do. Most individuals want to do construction, they're buying or selling houses, they need to get insurance and mortgage. So there's a large opportunity ahead of us. So there's mostly it's about scale, it is about understanding is where we are going to actually invest a tremendous amount of effort. And then last but not least, is the platform. So think about it from a perspective of the more we invest in the platform, the more we will build the ecosystem of partnerships around us, the more the usage, and the usability of our of our acid, which is the digital twin becomes valuable both to our customers and and to the rest of the world.

Very nice. Man, the gigantic opportunity in front of you, you think about all of the all of the spaces that exist spaces under management seems like a very appropriate, wonderful metric.

And maybe one of the things sorry to add would be like we have this global opportunity, you can imagine one of the things that the pandemic has really caused is because of the lack of movement, you couldn't go somewhere. And as a result, let's say you had to start a new factory line, you can't actually go there, if you wanted to see how the factory was doing, you couldn't go there. So it's really made it important on a global basis to build a digital twin and use that for a variety of different use cases. Trying not to you know, state, but one of the things that happened over the course of this was there needed to be a new vaccine line built. And the the company that was going to build it needed a replica of what the original line looked like. And so they came to us and said, Hey, we need to build that line, can you get us scanned in a day. And so our, our team went in and actually scan the original line, got them the digital twin, and they were able to build it over, like the course of a weekend and get that started. And you just can't do that that efficiently and fast, you know, on a on a global basis. On accounting, you have those kinds of capabilities.

Yeah. COVID definitely highlighted the importance of being able to be somewhere without being there physically. And the opportunities that arise when you're able to create a situation where you can truly teleport in is profound, even if it's on a not real time basis, right? This this notion that you're capturing, weekly, monthly, whatever happens to make sense for that particular environment in their use case, there's incredible credible amount of power there. As you kind of have this tremendous amount of momentum behind the company. Are there are things that that concern you as you look ahead over the next 1218 months or so, is there something or someone in this industry that makes you nervous?

It's always a interesting challenge is like you don't know what you're you're looking what you're worried about, you might actually see it. I'm pretty good at, you know, sleeping at night. So it's not like a major issue. As I think about the future, I think the key is going to be around our understanding. So can we push our understanding more so and at scale? I think that's the other part because it's one thing to understand semantically, just a little bit of stuff. It's Can you understand that in context at scale? I wouldn't say I'm worried about it, but definitely a big challenge ahead of us. As I mentioned earlier, it's just it's a non trivial amount of work. It's not something that has happened yet? You know, most of the understanding that we do have in place today is pretty rudimentary across the industry. We call things AI. That's why I was like joking, like, AI is such a overused word. Now, we have very little understanding of the world as it stands today, from a machine learning perspective, and how do we push that boundary? How do we push that agenda? Even with our, you know, the large scale companies, we're relatively small company, but even with the large scale companies, it's very similarly, an issue in place today. So that would be like area and a challenge that I think is doesn't keep me up at night, but is certainly something that I think about on our on our daily regular basis.

Yeah. How do you think about that, and I asked that from the perspective that you, as you noted, had been spent time at eBay spend time at Google, we've been at Microsoft, you're recently CTO of carta, you've been a lot of companies where this challenge of having the computer understand the world or understand humans has been part of the mission. How do you think about this? How do you think through this process of building out this type of Development Environment and Development team that helps you solve this sort of problem? Because it's not quite like building other types of enterprise platform software?

That's right. A few key things. One is, it's a different breed of person who, who has to come in and do this, like it's not a, it's not for the faint of heart. This isn't a, you know, a statistical understanding of of things. A lot of search engine technology, you know, because statistics rarely make make a difference have have been useful. And it doesn't work in this context, with imagery, especially with computer vision, knowledge of of the graph, I guess I'm biased, I should say, maybe I'll start there. I'm biased in the sense that I believe, for understanding to happen, there has to be memory, there has to be some notion of a graph that you know, and understand what is contextually you know, an entity? How does it relate to another entity? And then can you walk a graph of associated entries? And pull that all together? So you're asking how, you know, I fundamentally, believe it, it stems from a knowledge graph. So do we have enough of a knowledge graph that can give a scene understanding? Or not? Do we end the knowledge graph has to have enough entities in there as well, to be able to understand all the various components that could be in a space all the way down to a vertical, a sub vertical and so on? So first and foremost, you know, is do you have a knowledge graph? Is your knowledge graph good? Was it trained? Well, there's other notion of smart data, which is almost all of machine learning and deep learning today. And again, this is my my point of view. So there might be right or wrong. If you talk to a data scientist, they spent an inordinate amount of time in trying to get the training data, right. And spending a lot of time doing the training, the network's the algorithms that are then you know, pushed on top help. And so I'm not saying they don't, but you know, the fundamentals are around that training data set. And then it's tweaking the parameters, the layers in the network, to get to the right, you know, outcome that you're looking for. I had a running joke with my team that almost always you could get close to the answer by using a few very basic networks that were actually not that CPU intensive. Use a, you know, off the pie, like a much more heavy duty algorithm, it will get you like a few more points of updates, but you could probably be better off training better in the first place. And so they trained it is that that graph, to me goes to the How can we build enough of that? Can we build enough of a context, and that's where some of the larger companies do have an advantage? Like, they have a lot of data like that's why the Google the Facebook's the apples, Microsoft's of the world, they spend a tremendous amount of time collecting data is because that's what gives them more contextual information about a particular subject, and then being able to use that for better understanding. So to me, that goes to the how, yeah,

very nice. From a leadership perspective, have you been an engineering leader across many of these companies? What's the people side of this puzzle? What are the sort of key ingredients to establishing an engineering culture that is able to deliver the sort of innovation that you need not just not just once, but some sort of continuous basis?

Yeah. Culture wise, I've been a strong proponent of capability and compatibility. So early on, I had the pleasure when I was at Google, helping build the Google analytics team. And it was what I was The two favorites and highly performing teams that I've ever built in my career, had the pleasure to what I learned there was that, you know, assuming there's innate talent, there's, you know, you've actually measured for that compatibility is also very important to having a team cohesion, having a team kind of work together common goal, you know, no internal politics from a cultural perspective. And then once you've hired this smart people, then getting out of their way, like, you probably heard that statement many, many a time. It's very hard to do. But once you do it, and we've got the right team, you've got the right talent, you've got the right compatibility, you've got a lot of diversity within the organization, then it actually works, in my opinion, quite well. And I'll give you a 3d side, quick anecdote. One of the best things about that team was with 107 people move. Google was in those days, like moving every three to five months, we had 107 people move offices, not a single person had the same first night,

hundreds of people, not a single first name that was the same.

That's incredible. It was incredible. And I remember like looking through the list and sourcing and going, how is it possible that, you know, there's hundreds of people moving, and we don't have a single same first name. And to me, that all just highlights, you know, when you have a awesome diverse team, and that strongly talented, and then you let them go, they produce some amazing results. And we did some amazing things during those days. And similarly, like, you know, we built the assistant, and abled our new product development team at eBay. Same thing, the team was just amazing. We, we built a Knowledge Graph barn on, we did semantic, multilingual embeddings, that allowed us to have multiple different languages in the same graph, we built a Knowledge Graph across many, many different areas that worked for audio video imagery, as well as text. So the ability for us to be able to do that in context is, again, non trivial. So having that kind of a team, that culture that promotes active thinking, diverse thinking, and pushes out great product, at any given point in time, is probably helpful. The adage of always stayed with is, you know, you come to work to have fun, you come to work to have fun with people that you can learn from, and then you trust, because trust goes a very long way in ensuring that they're doing, you know, the person's got your back. So it's not you're not fighting with them if you're actually working with them. And you know, building something great. You ship early and often. And it's you actually seeing the outcome and the results of your work. If you keep you know, a few of those outages alive, that the team just naturally culturally gets great.

You mentioned two concepts that might be perceived as somewhat opposing. One of them is diversity. And the other is compatibility. How do those things fit together?

I've never seen them as opposing, I guess, is maybe the first way to think about it. So to me, when you build a team, the the interesting part is finding that next person who's going to provide a viewpoint that might be different than yours, but in a way that is different, I guess, is the only way to think about it. But in a compatible enough fashion. Like it's not, you know, again, you put your ego out the door. I'll give you another anecdote. One of my colleagues, I just joined Google like day one. And just just to let you know, like, I've known this guy now for 17 years, he's a really good friend, we we hang out talk all the time. First day, he came on, and he's this very typical Swiss guy. If he hears this, he's gonna know who he is, anyhow, very difficult Swiss guy. And he had a point of view that was diagrammatically opposed to mine, like 100%. But there was zero ego there. It wasn't about, like his viewpoint was like minds The only right way, it was just his viewpoint. And between the two of us, we would have a conversation, and I can guarantee it like we would come back with a better outcome if we ever talked. So you know, since then, we've always stayed in touch. We've always talked, I have worked with him multiple different locations over the years. And invariably, he comes back with a different viewpoint. But we always work working out. We always have a great conversation. And then I believe the outcomes are always better, because we've actually talked about it. Very dogmatically Opposing Viewpoints style wise, we're very different. But we're super compatible in the sense that we can have a conversation we can be you know, check our egos at As I say, out the door, and come back with a better solution. So to me like its diversity and compatibility actually go hand in hand, if you get the right combination. You can also have the wrong one where you have somebody who just doesn't get along with somebody else. And they could be the smartest person on the on the planet. But that team is probably not going to function very well.

What do you do in that scenario? What do you do in this scenario where you have somebody that's truly brilliant, but not compatible with the rest of the team?

Yeah. Funny enough, this happens all the time. Like it's a, it's a, it's a conversation that you end up having with the with the individual who's, you know, truly brilliant moreso than the team, I always believed in the team. So I believe in the collective team, I believe that's the direction one has to go. So if you have one person who's truly brilliant, and the team's unhappy, not quite my cup of tea personally like, and those people who would argue that, hey, that works quite well, like we build a team around a person versus the other way around. I personally build a team. And so in that particular scenario, either the person has to shape up and become more compatible with the organization or not. And I've had situations in the past, like I've had team members who are superbly brilliant, and superbly arrogant, usually, and having that conversation with, you know, telling them, this is not gonna work. Unless they do it. Sometimes it works. Sometimes they they move on to another team.

Got it? I think that's a battle. As you noted, it comes up frequently for you. But I think that a lot of Engineering Leadership or company leadership, they struggle with this one, how do I deal with the brilliant jerk? How do I deal with the person who, on their own is amazing, but it doesn't work well with the broader team. And so thanks for sharing your perspective on that one. One of the things that I as I was getting to know you, one of the things I thought was really impressive was this personal challenge you'd set for yourself in your own continuing effort to become a better leader, a better manager, better performer in this kind of pursuit of improvement? You set the goal of becoming a polished keynote speaker? What motivated that personal challenge? And how did you follow through?

Well, first off, I don't know if I ever became polished. But But yes, I did put in a challenge for myself and said, I wanted to do kind of a large enough audience keynote speech, you know, over the course of a year or so, as a personal goal. I've typically been super shy, like, I do well in like small team events and such. But over the years, I should say, I've had to, like, you know, be able to come out and present at larger and larger internal and external audiences. And so this one was just a, hey, we've built something super cool. No one's actually talking about it. And instead of like, trying to make somebody else go talk about it, you know, how about if I take on the challenge on a personal level, and then, you know, build, build my way out to a large enough audience. And I'd set a goal of like, 2000 folks, or so I wanted to go to a keynote speaker that was going to have, you know, more engineering type 2000 keynote audience. And the, the fun work actually happened. Like over the course of that, I got more comfortable, I started small, I started with, you know, 50 people, 70 people, 100 people, 400 people, like, you know, like, started at various things. They took pretty much every gig I could to speak because to me, it was always about practice, knowing your content, like if there's I guess one thing I could tell folks is, if you know, your content, the nervousness quite vanishes, because you show up. I did, you know, keynotes, I did firesides, I did panels. And as is always the case, like, you never know what someone's going to ask you, if you're in a panel, or if you're in a, you know, in a fireside, you know, you might have some hands, but you, you know, they might actually come up with something new along the way. So you have to also be able to think on your feet. So you need to know your content. And I was like probably the biggest realization that came through all of this is like, if you're confident in your content, if you know what you're talking about, then showing up to the event is actually not that hard. It's just a little bit of prep work. And then after a period of time you end up you know, having enough experience that you can take any incident into account. And by incident I just mean like, as part of this journey, a team invited me to come out and speak at a conference that was about, I think, 1500 people or so, in Peru, and the first 20 minutes was a keynote the next 20 minutes with a fireside and the third 20 minutes with a q&a. So you have no idea what you're going to ask you're in a completely different country and trying to answer their you know, questions that they have in a very different mindset that they have as well because they are not living in Silicon Valley on a on a daily basis. Answer the work that folks are thinking about is very dramatically different. So how do you answer those those type of questions? Anyway, knowing your content was probably the biggest realization that I realized I ended up with. And then once you did that, if you knew it well enough, you'd experience enough, then it felt like it was a much easier journey from their own and fine and fun.

As you were preparing as you are getting to know the content, what was the strategy to really get to know it? Did you kind of script out your keynotes and then work on them enough? So you kind of really internalize that information? Or do you just take some sort of other approach mind mapping approach? What was the technique that you use to really master the content? Yeah, it

was definitely working, you know, building the content, building the slide deck, changing it every so often. So you know, I would say that every time I went and presented, I probably changed between five and 10%. The content, you know, looked at what resonated, what didn't, and then come back and update it. Definitely, in the early days, scripted a lot, you know, every slide like here are the talking points per slide, practice those, and there's just enough practice. Again, it's different styles for different people. For me, I can't practice too much, I get too rehearsed and feels awkward. So just enough practice that, you know, feels more natural, and then starting know that a lot of internal presentations as well prior to that. So one of the things that probably helped was, you know, just within the team, talking about different things, having that conversation. And again, getting to know your content well enough, was helpful. Maybe the last thing I would mention is panels and firesides are actually great practice. Because what they end up doing is when you're answering you know, in a much more, you know, it's not a scripted fashion. And so you're actually answering them in a much more genuine, authentic fashion. And what ends up happening is if people ask you questions, or the audience asked you a question on the panelist house few questions, you end up learning a lot more about what really matters to the audience, or the, you know, the folks who are listening to you, maybe that'll be the second nugget for things that I learned was presenting is not about telling people what you know, presenting to me became about telling people what they want to know, and trying to really understand what the audience wants to hear was the fun challenge. In some ways. It's like, Okay, I know some things I know, my part of content, I know where, where I'm going with this. But what is it that the audience really would like to know? What what are the questions that people mostly ask and such, and then trying to tailor that, to me was really interesting. And once, at least, I thought, I figured that part out, it became a lot more fun. And the reviews I got back, were better. Because now you were trying to understand the audience and where they were, you know, especially if there was a marketing audience that was there or more engineering savvy audience or, you know, the cxos of the world who are in there, they had a very different mindset of what they wanted you to present to them, how much of it can meet them at the level that they are of understanding of that particular topic. Because if you start citing, you know, neural network types, to a marketing audience, it's probably not going to go well, they'd be like, okay, that's, I have no interest in this. Or if you start, you know, giving marketing stats to a to a technical audience, then they don't care either. And so learning exactly what they wanted you to tell them or what they wanted to hear, I think was another insight that was interesting. For me.

That's great. A lot of hard lessons learned. I'm sure.

She kind of went through those fun. Yeah, there was some, you know, moments in time they're like, are like, Oh my god, what am I doing? I remember, like before the largest keynote piece, I remember, like waking up in a cold sweat at 5am going, what am I done? What's going to happen? I don't think I'm ready. And then and then it actually went off, upgrade. So

that's nice. That's great. Let's wrap up with a few in lightning round questions here. What commonly held belief about spatial computing?

Do you disagree with I'm waiting for the day that I can get glasses that helped me to me is it like only one thing, like, you know, is only the glasses are going to be like, GPU sitting in a cloud? Is it you know, GPU sitting on my phone? I, I'm fundamentally again, of the belief like, we're going to try and use everything possible. Why would you like there's no like one panacea? I don't believe so. I'd love to use everything that they possibly can and then make that a thing. So

yeah. Besides the one that you're building, what tool or service Do you wish existed in the AR market or in the digital twin

market? Air market? I think you and I chatted about this earlier. I want a lens in my eye. That gives me active information. That would be great. I'm ready. I will. I will install it at any given point in time. So let's go I can dabble every so often in the early version of like neural link, there's some things that you can do. They've built some headsets that can let you navigate and answer a question such just by thinking. They're great. Like, I want them to get better and better. I don't know if I mentioned this to you, but I'm a futurist. So I fundamentally believe things will get augmented in really interesting ways. Over time, so bring it on. Amen.

Very nice. I'm with you on that one. So what AI is, is augmented intelligence, fundamentally, another interpretation anyway,

I love that that's a good, that's a good good one, there was an IBM study that mentioned that the amount of information by like 2030, or 2040, you're going to have doubling of information is going to increase at an exponential rate. And so the amount of information that's coming into the market is just going to kind of continue to double and double at a much faster pace. Right now, it's, I think, they said it was a year, you know, by 2040, might be a day and so on, or whatever the timescale looks like. And so the point always was that, you know, 100 years ago, you could be an expert in in 10. Fields. Today, you can barely be an expert of like, you know, a subsection of a sub section of a subsection of a field, because there's so much information that you have to know about that particular area. And so, you know, 10 years from now, if we don't have the augmented intelligence, to your point, being able to sift through the right level of information and understand that it's just going to be non trivial, non trivial.

Yeah. If you could sit down and have coffee with your 25 year old self, what advice would you share with 25 year old job?

For one, I consider myself still 25? No, I'm kidding. They think thing that I think comes to mind is, you could do a lot more, I guess, is the way to think about it, you know, experiences shape your life to a large extent. And now I realize some of my experiences have shaped me over the years. But even at that point, I could have probably done a tremendous amount more, I just moved to the US about that time, but a couple of years prior, and he's still kind of figuring out navigating the country. And so I would say that, you know, I was probably hesitant to take on any major life changes beyond just this kind of massive life change I'd done. But I could have probably done more. So I guess is the thing I would say, you know, I think capacity to do more. And so that would probably what I'd say,

to push yourself, encourage yourself to Yeah, what book Have you read recently that you found to be deeply insightful or profound?

It's funny, my wife and I joke about this all the time, she's the one who actually reads for profound books I read usually what I call crap, you know, novels, mystery, and so on. The thing I do probably listen to is like, I listen to a few podcasts, I listen to radiolab, a recent listen to the knowledge project. And there's some always interesting things in there, mostly because of the vast variety of things that they talk about. So that's what I would tell you, like I get excited by one of my favorite podcast was about how forests work. So it's a completely different mind space. And the fact that different trees in the forest actually keep it together in ways that prevent disease, destruction, how they communicate, navigate talk to each other, is, is a fascinating way to think about, you know, life. One of the things that probably doesn't come across, I'm an avid tree hugger as well, I think about the environmental at all times, I sit on a board of a nonprofit environmental agency, mostly because of I do believe in actively in climate change. And I think about like, you know, how do we leave our environment better for our kids? And that episode on trees really got me. So if I was a fascinating, fascinating episode, there was another one on triage as to how you do triage, when you're thinking about, you know, if you're in war situation, or nursing, or as it's happening today with doling out COVID 19. Vaccines? What? What triage, do you apply, if you were to, you know, decide who gets the next vaccine, especially when you have a limited supply? And how do you decide that? And what was fascinating was, if you put 50 people in a room and ask them to come to solution, they all disagree on what's the right approach? Yeah, that was a really mind, mind bending conversation. So

50 different opinions out of 50 people in that way.

Yeah. So how do you decide like, who gets it?

Yeah. Any closing thoughts you'd like to share?

No, this has been great, hopefully, helpful and more than happy to answer any other further questions about matterport or otherwise,

appreciate that. Where can people go to learn more about you follow up with you learn more about the efforts here at matterport matterport.

We actually You're pretty active. You can see us on online on metaphor. COMM we just published as you mentioned, our IPO and spec deck sits off of our IR site. You know, feel free. Happy to answer any questions about that. me. I don't think I'm that interesting, but I'm sure you can find me by doing a quick Google search.

Awesome, Japjit. Thanks so much for this conversation. I really enjoyed it.

Thanks.

Before you go, I'm going to tell you about the next episode. in it. I speak with Amy LaMeyer. Amy is the managing partner of WXR Fund where she invests in early stage spatial computing and artificial intelligence companies with female leadership. In this conversation, Amy shares her perspective after attending the recent Y Combinator Demo Day in startup events at South by Southwest, we talked about market trends, the venture fund some specific companies they've gotten excited about, and some advice for entrepreneurs. I think you'll really enjoy the conversation. Please follow or subscribe to the podcast so you don't miss this or other great episodes. Until next time.