How to Use Customer Feedback and Data Analysis to Iterate Your Product
9:21PM Sep 23, +0000
Speakers:
Keywords:
customers
feature
product
api
stephanie
people
users
plaid
ebay
roadmap
feedback
data
spotify
envision
set
pandemic
important
startups
changed
partners
Hey, everyone have a great day here to discuss how to trade your product. And it's the third time we do this panel. But I think it's particularly topical this year because I mean, data analytics are more important than ever. And this means that startups need to balance this versus customer feedback and many other sources. So that's what we are going to discuss with our great panelists. We have Johnny gray is from Plaid, who is going to give us his future beer perspective. We have Stephanie, Mike Hurley, from indivision, which has another layer to the topic, because it's also used for people to iterate on design and product. And we have Pete Thompson, from eBay, which obviously is a really big and very diverse customer base. So I'm actually going to start with you, Pete with a question and data. And my question would be, could a start up or a tech company ever be true that that driven?
Me not a trick question, I would just say the answer is absolutely not. It's a matter of how you use the data and how you balance it with with other forms of feedback that you get. But I would say absolutely, I mean, it's just getting more and more important to uncover things within the organization with data that that you can't do with any other means. It identifies things that sort of human curation or manual processing wouldn't uncover. But at the same time, I think you need to be able to also think about sort of other things that you're not doing, or, or features that you could build that the data won't actually tell you. Just to give you a couple examples that eBay, so we have a, you know, global shipping platform. And we're constantly trying to figure out how to tell customers what their estimated delivery date will be, especially because we don't control the all the logistics, and we could be shipping something from Sweden to the US. And so we have hundreds of algorithms that are constantly checking things and looking at all different signals. And we were last year, we uncovered that we were actually, you know, about, you know, over 20% of our packages were actually arriving earlier than what our data was telling us they should have arrived at. And so we were able to, through that, we were able to get some amazing insight and actually set the expectations when people are out shopping on the site, that they'll get the product much faster. So that's just an example of how data can can do something that humans would never be able to come through all of that. On the other hand, we've recently been launching something that we call it searching through images, so not just text queries. And this is a great example where our data would never have told us that the younger audiences just want to be able to, on our site, want to just be able to browse and see, you know, if they're looking at fashion, they just want to see other things that look like the same image, not based on sort of a text based search. And these are things that you have to glean through other forms of feedback loops.
Great. Well, I'm going to ask you, Stephanie, if there were ever instances that that help you find things that you didn't suspect your user wanted.
Yes, definitely. And just to echo everything that Pete just said, data is important. I will slightly disagree maybe and say that there is a point where you can get to data driven, where you're not seeing the what I call the the edges of the innovation bell curve. So it's really important to see what smaller cohorts are doing, because they could be early adopters to a new behavior. And that's something that that we frequently look at and envision and in previous labs for myself. And a lot of times when you look at those early adopters or behaviors in the data that you're seeing that are smaller groups, and you make sure that you're driving features that are driven with hypotheses behind that, you might not get it right all the time, because it might not be an early adopter. It might be someone who just does something really different, like we see at envision, and the way that people work, that everyone is a bit of a snowflake, right? We all have our different workflows. We are all unique humans, and it really matters, the team that comes together and the goals that they have and the intricacies of how they work together. And so we're finding at least in our data, that it's okay that everyone has a unique snowflake, we need to create things that are malleable and are usable that people can like make into their own as part of their workflow.
Right well They'll actually echoed something that Johnny told us when we're preparing for this panel, which was about the concept of representative customers. So Jenny, could you please explain that? And also, maybe tell us how you select those users? Because I'm sure it's not an easy process?
Yeah, sure. So, I mean, this goes beyond data, just because about how you approach you know, building products, you have you have hundreds or 1000s of customers that are telling you they want you to build, you know, different things. How do you? How do you go from that to, you know, a product roadmap, that makes sense. And I think something, something for us has been really important is to identify the customers who we think if we solve their problem, we solve the problems of a host of other of other developers. And the idea is to treat those customers we call representative customers more like partners, right? So the key though, is if you pick the wrong representative customer, you may only solve their problem and not a broader problem that applies to your wide user base. So you have to you have to be very careful. So I'll give you you know, I'll give you an example of one customer that we work closely with, actually over the last nine months this year, that's been very helpful. It's a it's a small PFM. It's not small, it's going quite fascinating, really well called copilot. They're an app to help you manage your finances. And well, we recognize that they were say finally more more support tickets per kind of, per unit of billing that they have for us. And we found interesting that we looked at the tickets, and we thought that their tickets were very sophisticated, meaning they had a really deep understanding of our product and how it worked. And it's contours and realizes that actually like that one customer we had was the best one to get insights from their end users for us, because they were willing to like push that information through us. And then we looked at them a bunch. And we saw that they represented a big category of our customers, maybe like 1015 of our customers have very similar use cases. And so you know, you had that you had a market mapping where you were like someone very demanding and sophisticated. And then you look at new stuff, like their needs, were very similar to a large segment of our customers. So you have those two things in common. And then the question is, how do you turn them into a partner, so that as opposed to just filing support tickets, right, they're willing to talk to you and help you develop the product roadmap, and that works super, super well for us? And, you know, the I really like actually, what, what what Stephanie said, right? When you look at data, or even like product roadmapping at all, there's the danger, if you have too much data is it looks like he's right. It's just, it's like the average of everybody and from the average is difficult actually, to build great products, right. And so you have to find a way to like segment your customers into little buckets. And some of those buckets are like, you know, potential trends that are emerging, like new sets of developers. But there's, there's, there's other buckets that make sense, right? Like market strategically for your business, you think you think three years, four years from now, we're going to be really big. So I really urge people like first, like, kind of narrow down your set of users, and then kind of looked at the data and then really go deep into the the kind of product signals that you're getting.
So please, I'm sure that the problem is quite extreme at eBay, because your user base goes basically from teenagers selling their own sneakers on eBay, to really large corporations. So how do you balance that in terms of feedback should take into account?
Yeah, it's a great question. I mean, just to put a point on that we have around 20 million global sellers across under 90 markets. And it could be high school students with that sort of, you know, a side hustle to a global brand, like Nike. And so it, they have obviously very diverse needs. But we are a sort of horizontal marketplace that that we need to be able to account for all of them. And so one of the things that that we've been working on is to not go away, you know, do the segmentation, understand the different needs, and build for those needs, specifically, sort of vertical sort of focus, but at the same time with an eye towards horizontal extensibility. And that's sort of the nuance or the sophistication that I think, especially an e commerce company needs to have the ability to really understand, for instance, a sneaker seller and what their needs are, but at the same time not have to do that across 400 different categories, but figure out how to find the archetypes how to how to find sort of commonalities, common components, that then you can figure out how to take horizontally and actually be efficient and meet the scale that you need to do. And I think it's an interesting combination. Like in terms of looking for product people. I'm always looking for people that have that domain knowledge, like in retail, but also come from a tech company, where they have had to think about really scaling the technology. And that's a great intersection, too, to have that skill set and capability.
Right. Before I forget, I just wanted to remind the audience that they can ask questions, so you can ask questions in the chat and we'll address them if we can. And I also wanted to ask Stephanie, her perspective from a freemium product. How do you balance free versus paid user and then funnel in terms of building that's it process?
Yes, definitely I've worked on many of freemium to pay in product. So there's definitely a strategy behind it. I personally love to use metaphors when speaking about these sorts of things, I think of it like a pool that you fish out of right, and some fish will always stay in the freemium, and that's okay. But it's all about that virality and making sure that the things that you are creating have enough value to punch you into a paid service. So like Pete said, it's very unique depending on the customer cohort, but how we build things that are extensible is definitely what we're focused on at envision. So say, a workflow and the way that teams collaborate, we see through research that there's a, there's kind of a baseline of how teams collaborate, right? You come together, you go away, you come together, you go away, whether you're a design team, a product team, in HR marketing, you work at a bank, it doesn't matter, right? And so we are really focused on how do we bring those collaboration moments together through these, like, maybe someone always needs a timer, or maybe they need a poll, and it's like, doesn't really necessarily matter which cohort you're in. However, each of those needs of those individuals is unique, right. So they might group a set of features together differently. And that's where the value is for them. And that's a big piece of how we market our product as well
understood and paid in our conversation. I think he also mentioned SMBs, as a big segment. So I just wanted to take a step back from what we're discussing, that sounds more like personas, and really talk about the type of users that a company knows they are prioritizing in the in that process.
Yeah, I think Johnny and I were having that conversation before. So feel free to jump in. But I, you know, I think when you're focused on sort of different categories, or different segments, one of the things that we've tried to do is also lately, launch things that are sort of API first products. So you're working with a small set of third party developers that then have additional customers that they talk to, and they have domain knowledge that you may not have about what those customer needs are. So as opposed to, you know, we just recently launched the ability for all of our sellers to put videos on, on their listings. And again, that that's a very different requirement if you're if you have 1000 listings versus one listing. And so we worked, we decided to launch it first as an API. And typically you build it for you know, you build something first party, you, you launch it, and then six to nine months later, you figure out how to launch it through your API. So flipping that around, enabled us to really work closely with these partners. And to figure out how do we scale it amongst the SMB market that they were sort of, you know, that they are channel partners for, and it turns into a great way to really get the nuance and and specificity that we were looking for in terms of the feature set?
Well, it sounds to me, you know, a thing or two about API's. So I wanted to ask you about how that impacts their iteration speed. Because we often hear that startups should move fast and break things. And I'm sure, that might not be how you see things at Plaid. So
I'm, yeah, so I actually just to go back to that, but I think the lens to what to teach us that because it's very interesting, what he said was, Hey, he's working, you're building API solutions to to a customer who themselves has domain expertise. And I think that's one of the advantages of building an API company like, Plaid, we're not really in the business of predicting the future of consumer finance. But there's all these experts that are building on top of us, and they're pulling us in that direction. And so that's, that's like a wonderful place to be. But the disadvantage, I think, of building API solutions is the timeline is very extended, right? Say it takes you six months to build a feature right on an API and you've built it. And then you have to convince your early partners to integrate it with the API. And then it takes them six months to build their feature on top of that API before they see if they have product market fit. Right. And so your development cycle effectively is it's Oh of the length of your development cycle plus the length of the development cycle of your customer. And that's a real, that's a it just, it just means you have to be they have a different approach to how you build things. You have to have higher conviction upfront, because you can't just AV test your way to success. Also, by the way, if your customers are using your API, you can't like deprecate it. You can't iterate on it in a way that breaks that because for them to switch to the new version requires product and engineering. And those things are very expensive. So I think for Plaid, we can't say, I mean, the first because we're in financial services, you can't really have the break things like that's just not as a no go when people's mortgages depend on you or, or money moving around. But moving fast is important, just moving fast, looks different, right. And it's relative speed. It's not we're not moving fast relative to Facebook are much slower than Facebook, we're moving fast relative to other API companies. And I think the key there is you, you need to either have a lot of design partners, that is customers that are willing to work with you on beta things, and they're willing to give you the feedback and are willing to have constant engineering time dedicated to working with you, if you don't have that it's, it's not going to work. Or you need to find a set of customers with whom you can iterate. And for us, one set of customers, we found, it's easier to iterate with our self serve customers, because of what a self serve customer does is they they sign up for your API, and look at the docs, and whatever the docs are, that's the reality of the product. And so you can you can ship small things with those new developers and get quicker feedback with one giant caveat, new signups may not be representative of your entire user base, which goes back to what Stephanie was saying at the beginning, right? If you, if you pick like a sub segment, and it's not a trend setting, then you're going to kind of get in trouble. But, you know, for us, it's like looking at things that we can do building API businesses that can still have a lot of speed. So that's about product conviction, design partners, or finding something like self serve customers that you can ease into a new experience without, without a lot of lift necessary on both sides.
So Stephanie, please correct me if I'm wrong, but I think one of the envisions purposes is to help customers be fast or their iteration process. Is that correct? Is speed a big part of it or more efficient?
Yes, I actually think it's, it's all of those things, right? speed and efficiency, but now more than ever, it's about inclusivity. So we are very, at least in our research, we're seeing that one of the silver linings of the past, say 20 months is the amount of inclusivity that teams now have for one another through a lot of the products that we're creating. So the ability to give feedback to one another to do that asynchronously, or to not necessarily have to be the loudest voice in this virtual zoom. Right. And so we really believe that internally to envision, we work in a way that we hope is the future of work, since we were remote before remote was cool. Um, and so, you know, I always I always am so enlivened by our product, kickoffs and our solution reviews, because we use these massive free hands that have everyone stickies and feedbacks in it. And for the most part, sometimes I don't even say a word, I just add my feedback in there, which would be like unheard of in previous roles. And it just it like I said that inclusivity, for me is actually the key differentiator and the silver lining that we're trying to just 10x through all of the features that we're creating. And I think that is truly what is going to be changing our industry is the ability to hear more voices, and more ideas to spur that inner innovation, whether it's you're allowed on zoom, or you're visually collaborating and making together.
And that was also the sense of my question and free versus paid users, because sometimes we hear that so free users are the loudest, and you may want to listen to different people. And I was curious how, for instance, do you balance? That's, I don't know, segments, my majority. Do you think here most people who send feedback are not necessarily the people you want to listen to? Maybe story starting with you, Stephanie.
Yeah, definitely. So I, previously before the pandemic boxed, and so don't get me wrong, when you get like a big hit to the base, it hurts. So some feedback hurts. But that feedback is valuable all at the same time. And so really, it's about how you and the team synthesize these things and take it from something that maybe might seem like a negative and turning it into something like a positive and what do we actually do about it? Yes, there's a lot of noise and it takes ruthless prioritization. We always tend to prioritize against like a variance of metrics, but also as a, what we call a four legged chair and envision so we have product design, engineering and insights. And so really, it's there's a science to it to understanding what's the reach of this, you know, complaint or user problem? What's the ROI on it? How much money would we potentially make from paid people? Don't get me wrong. But then also, it's kind of this dance between the four functions, right of saying, Oh, this is a problem, but it'll take us three quarters to fix that well, or we could this is maybe ranked fourth, but we could fix it in two days, let's just go ahead and do it. And so it's that not to be super meta about this collaboration topic. But it's also that we're that requiring of that, no longer three legged stool, but four legged chair to say this, Beth, based on all of our expertise is what we believe in is best for our customer and best for the business. And so it's a little bit science and a little bit just getting together and, you know, putting the best minds towards things. But that's our approach and envision anyways.
Okay, well, there's actually a closer question from the audience, which is, how do you communicate with users or a group of users that feature there? Oh, maybe Luckily, requested isn't something that you're ever going to add? So I think, yeah, Johnny, you have a bit of experience with that. And then exactly that. But in the beginning of ignoring customer feedback on purpose, can you explain it?
Yeah, sure. I just hope this doesn't make its way into like, the Twitter or anything else, no quotes about anything that's in the next couple of minutes. Yeah. So in the early days of Plaid no longer but in the early days of Plaid, it actually felt deeply like in product and engineering and design insights, like we were, it felt like we were ignoring customer feature requests. And, and, you know, I came from from a company that was very customer centric. And so it was a little weird. When I applied that I had this emotion, I had this feeling. And I was like, kind of trying to figure out what was and what happened is we had, we talked to a lot of customers, and they all said what they cared the most about was conversion, right? Because Plaid is in the onboarding flow of all these apps. So so they cared most about conversion. And if we could increase conversion by 10%, we would make 10% more revenue, and our customers would have 10% more users, and hence would probably make 10% more revenue. So everything was happy. So that was like the number one thing. And then every customer came to us. And they were like, we would like this feature, we would like to clean data in this way, we would like for you to integrate with this, we would like the dashboard to have this feature, we would like your billing to be predictive, like all this stuff. And frankly, we almost none of that, because well, we sat back and we looked at our strategy. And we looked at actually what mattered the most to our customers, it wasn't the feature requests, it was conversion. And you could always say like incremental engineering and conversion would drive more business value for you and for your customers. And the way we knew that was true, by the way, it's one of our customers tried us against competitors that had the features. At the end, they looked at the bottom line, and what they cared the most about was conversion. And so I actually, you know, thinking about saying no to customer. So this is what's happening. And honestly, we just told customers that we weren't going to do the feature, we were just like, we're like, it is not in our 2018 roadmap. It is not in our 2019 roadmap, we won't say like never, because you never know, right. And at some point conversion stopped mattering as much right? For some customers, they were mature, you know, they didn't want the incremental user because it was a lower quality user. And so then those features started mattering. So you know, you never say forever. But at that point in time, you could say we're not going to do it, but what we promise you is we'll have the best converting experience against our competitors. And I think we can agree that's actually the most important thing for you. And, you know, the person who's filing the ticket or the immediate pm does not make them happy, because locally within their team, they really want the feature. But if you look from the leadership standpoint, at our customers, and what mattered the most to them, it was that thing. And so I think that's what's really important when you say now is you got to explain why. Right, and the why should resonate with them, because you're building a product for your customers. And so they got, they should be able to understand why you're prioritizing something else. Right? If over and over again, you're saying no, and you can't rationalize it. And you can't rationalize it for any for any segment, then then you're in trouble, then you're just not listening to customers. And but you know, by the way, the rationalization might just be like, Look, our margins aren't high enough that we can support that feature. Like it's just, it's the reality that can be lots of things, but customers will will listen if you explain it to them, but you got to show what you are doing for them. And you've got to show them why that's more valuable, right? Because that's why they're using it at the end of the day.
So it doesn't resonate with you or is eBay at a scale where it can afford to do everything.
Now, you should see our backlog. Um, it's all about good backlog management. No, I just a couple things I would add, there's so many interesting comments that people have been saying. One is to me like depending on the business you're in, so at eBay, and when I was at the Xbox business, you have such passionate employees that are actually the power users also and allowing that their voice to be heard. And especially with the engineers carving out enough capacity for them to work on things that they want to work on, is actually a great way to be a different version of customer centric. And so I, you know, one of the things we did when I got here at eBay is basically just carve out 10% of everybody's time to be able to work on what they thought what they wanted to fix on the site. I mean, they've been here for long, you know, for a very long time and have a ton of passion. And we found that that was really interesting how it changed some of the prioritization of the backlog features. Now, that's one cohort of feedback. On the customer side, what we've been doing is really focusing on trying to do more sort of ongoing practice, it's it, you know, you can say you want to be a customer centric culture, but it can be easy to kind of walk away from that and get focused internally on planning. So we actually have with our sellers, we actually have panels set up every Wednesday. And teams have to basically get on a calendar to be able to go talk to them or show them prototypes, and get feedback. And we we constantly circulate the panels. So it's new feedback that we're getting from different countries and different types of seller cohorts. And that's been one way for us to just get that qualitative feedback, and just practice and then just engaging. You know, I think Stephanie said at the beginning with the snowflakes and stuff, like he had some really interesting data when you just get it at a qualitative level. And then you can go test it with more experimental data, but but it's the right way to figure out something that you may be blind to.
I think it's really interesting for startups to realize that to eBay still does that. So it's not a real problem that only startups have. But talking about a problem that startups have, there was another question from the audience, which is, was asking, should startups in the beginning really narrow down? The audience they're targeting and the type of users they are listening to? And I think he may be more relevant for Plaid, because you started out early?
I don't think that I don't think there's a generic answer that one, like, you know, what you're trying to look for is a set of users who are real champions for your product and really excited about it and where it solves a deep pain for them, I think the more you're narrow, the easier is to find a set of commonalities that you can build for right. But the danger with the more you narrow is you're not sure if the commonality will jump to kind of bigger segments. And so I'm not saying it's like art more than science, but I think it very much depends on like the the kind of startup you're building what your customer said is, and and whether you're really confident that you can take hypotheses from a small set and expand, you know, to a broader market. And well, you know, if you look at the history of startups, there's a bunch of startups where I think Paul Graham always says, it's like, it kind of looked like, like a toy at first, right for some sub segments somewhere. But there was just this, this is this broad market trend behind it, which is what eventually makes it go, you know, mainstream and global and huge, you know, deca corn or whatever you want to call it. And, and there's plenty of businesses that look like that. And that's great. But there's also plenty of businesses where that's not that's not really like the approach that worked from day one. And you didn't have this undercurrent that was benefiting you know, I Plaid, we're lucky we started in FinTech before there was a ton of FinTech. And so we grew with the wave, right. And, and so, yeah, we only focus on a few customers a few segments. At first, we were definitely you know, we're partners to banks now. But beyond those, they were not our customer group at all, it just just happens that the world has changed. But no one would have predicted that when we started, right. It's like, that's survivorship bias. So I'm sorry, if that's not super helpful, I think you have to be deeply analytical and strategic and understand your set of customers. And you can make some hypotheses about what might happen in next year or two. But you're not going to know if your wave is going to carry to be huge or not, right, you have to have some conviction and love the space that you're in love your customers, and you'll see how big the ride is, in the end.
Everything is totally fair points. Go ahead.
I was just going to add on to the something there whether it's a new company or a new feature, or a new product line within a company like I I would say yes, you should start out very focused, and and very specific, but then it's a mindset, it's about being curious. And just being open to the fact that you you know kind of let yourself wander with what you're seeing with the customers and and be willing to potentially change your business model, change the product set change whatever it is and but but I would say you need to start with a center of gravity with something that you believe in strongly based on some insight. And then but I think the Curiosity is super important.
Right? Well, that actually reminded me of something I read about in InVision, which is this concept of product centric roadmaps. So as definitely get it. Can you tell us what that means? And how that works?
Yes, definitely. So right there with up it's it's the Curiosity is key, but also something that I echo quite frequently with my teams is how do we get bias busting in there as well? How do we bust our own biases that we might have for how we solve for our customers. And so taking that into solving for a roadmap, I too have had the weekly kind of evaluative programs are constantly speaking with customers, when I worked at Walmart, in the heat of the pandemic, we weren't even showing them necessarily prototypes, we were just checking in with them. Because we saw our market penetration obviously explode. And our customer base completely changed. It turned to a much older demographic. And so that was very humbling for us on the product and design and engineering teams, because our our apps were not easy enough to use for this market. And they were feeling, oh, I'm just stupid. I don't know what I'm doing. It's like, No, no, no, we're not doing our jobs well enough. And but what are their needs from us? And what are what are the things that we could simplify for them was most key. So that is what ended up going into the roadmap. But as far as envision is concerned, and how we formulate our product roadmaps, we also have beta group. It's one of my favorite slack channels at work, where I just get to hear from people how they're using it, and how their workflow has changed or how they want, you know, that different feature that we spoke about earlier. And so it's a mixture, right. And I think it's being really explicit around, which are the pieces in our roadmap that we can ship immediately. And so for us, speed is very important. We have this concept of slices at envision, where we just love a good pizza emoji pizza party. So everything is pizza is all over the place. But these slices are intended to be a grouping of features that bring value to a defined customer cohort, instead of just shipping feature one, feature two, feature three, it's like, we need to tell a story about how these all come together. And that's how we formulate our roadmap. But in the smallest of chunks, right? So just enough, is the key here to then hear what the market is saying here, what our customers are resonating with because, frankly, for us in helping people collaborate in their workflow, it's changed drastically, just like ordering groceries changed during the pandemic. And so there's a lot of unknowns right now. And for me, that's super exciting, because it's changing daily, right? And so speaking with our customers, through many different channels is key here. But then keeping that roadmap in play, where you have these, these slices, essentially that deliver value, because they're a group of features, not like a singular thing. And so that is what creates meaning in our roadmap.
Code. Well, I like the pizza analogy, for sure. And you talked about shipping new features. So I'm curious, Pete, how do you ship new features? And more precisely, someone in the audience was asking, if How do you decide to reach a certain group of users you agree to show a certain new feature? Do you do you send it first maybe to partners? Or do you just roll it out? Or do a B testing? What's your recommendation there?
Yeah, I think it's, I mean, I think you have to, I always think of like, college strategy development, but like a staircase, right, you got to work on both sides, you got to, you've got to go up incrementally. And that's really, you know, asking your existing customers what else that they would like to see and start working up the staircase. But you also have to create a way for you to jump to the top of the staircase and look backwards, and try to figure that out. And then and then hopefully, at some point, you meet in the middle on some of these things. I rarely do. And that's why you have a backlog. But But the idea is to sort of make sure that you're not, you know, you're looking at multiple horizons. So you don't get stuck, you know, either looking too much in incremental innovation or too much it bold sort of step function, innovation. You know, just just one aside on sort of customer centricity and picking features is also how you work with partners. When I was at Sonos, and obviously, we wanted to work with Spotify as an amazing music service company. But we really didn't know how to do that. And there was sort of religious on both sides as to, hey, we want everybody to go through the Sonos app and and, you know, control their, their music throughout their home. Or, you know, the folks at Spotify were like, hey, people come through Spotify to discover their music, right? And so we ended up never doing anything together until we sat down together and got the product people to actually start with let's just create cool customer experiences and let's just prototype And figure out and see what we could do together. And that's what really came to life then with this idea of being able to, you know, direct sonars through the Spotify app, and people were so concerned on both sides about the business model and about whom the customer and it ended up what happened is it, it bergeon, this amazing amount of latent demand that we that neither side really knew about, and it allowed both companies to grow. And so I just bring that up as another dimension for us to think about in terms of being customer centric and picking features is, is how, what is the right way to, to work with partners, typically, its business development, and you know, you start out with Win Win ideas, and then it quickly becomes win, lose. And then you spend a year basically negotiating down to the finer point. And I just love this idea of starting with customer experience, experiences, and having sort of experienced milestones that you are together looking at doing.
So Stephanie, I see you were smiling. Were you involved in that personally? Yes, yes. Pete,
I think we overlap. So I worked at Spotify on on exactly that the Connect feature that powers Sonos, then I just all of that really resonates with me, because it's something that I take through to this day is just create, let's make the thing, let's see what is possible. And then, you know, discuss debate, whether it's employee engineering, or partnership implications, but I really, truly believe that there's a power in not only building that customer empathy, but then designing something for them that sings that everyone's like, man, we got to figure out a way to make this happen, because you see it and you're not just talking about it. So yes, I was wildly like nodding and laughing. So the It's crazy, small world.
Yeah. It's funny. Yeah, we have a question from the audience, actually, because we say you should be a debate. But what do you do when your users are actually silenced? What are termed best ways to get users to give you feedback.
I can jump in on that a little bit. And then others can jump in, I would say like, just because they're silent, doesn't mean they're not telling you things. Obviously, if you have the right instrumentation, like on an eBay platform, we're obviously looking at what everybody does. And so and constantly understanding where they're going throughout the site, what they're how long they're staying on things, what they, you know, start and then walk away from what they complete. All of that stuff, you know, should should speak loudly. And so even if you can't, for whatever reason, talk directly and, you know, verbally with your customers, that doesn't mean you can't figure out a way to to get those feedback, loops, you know, through an instrumentation standpoint, and I would highly encourage trying to get both because sometimes they'll tell you things, but their behavior is different. And so I think it's important to be able to have both mechanisms.
Right, then percent isn't this to go ahead, Stephanie?
Oh, sorry. I was just going to say there's also a difference between customers who are silent, and customers who just love your product no matter what you do. And you're like, no, this is not helpful. So like, this was the case at Spotify, when I was living and working in Stockholm, Sweden. And that's where the headquarters is. And many Swedes either know, Daniel Eck, or they had Spotify since the beginning. And we had this beautiful lab to test things. And against that was like this retro looking living room. And it was just a dream. But every person that we brought in, was like, yep, this is amazing. Like, we could have shown them a prototype that was all black screens with like one letter on them. Like, yes, I would know, this is amazing. Spotify is amazing. Like, okay, we have to do something about this. And I think it's, it's relevant, especially now, for a time when we're all tired. Our customers are probably tired, our users are tired, we're not getting the best insights out of folks right now. Because all of us are great, still reeling from COBOL pandemic. And so it's, it's, it's necessary to get scrappy, right to hear from people. And to not only necessarily, like, not just listen to what they're saying, to read their body language over zoom is also a lot more complicated. But the way that we handled it back then at Spotify was, okay, we have an international audience, we need to like just quickly see if some of these things are resonating. So what we did was we went into hostels, all around town, because that got us a more international audience. We could buy them a coffee or give them you know, a month free of Spotify. And like that kind of scrappy guerilla research or whatever you want to call it. Just find a way to speak to folks at envision. We even just use in our internal groups like, hey, finance, how are you? How are you doing with your workflow, and how are you using freehand and like, I can't I can't tell you the amount of surprises I've seen, just by Speaking internally to folks who are in marketing or finance or you know, just hearing from them because they they're not in in the weeds in the trenches with us in the day to day and how much it helps, again, bust your bias.
Well, I'm sorry, we have to end the panel now. But based on the conversation we had, I'm sure there will be a fourth edition next year and hopefully customer feedback will be back up again after the pandemic Fingers crossed. So thank you so much everyone for for green sites and everyone who joined us.