The Humans Who Power(ed) Your Algorithm: The Untold Story of Big Tech’s Curation, Content, & Editorial Teams

1:30PM Aug 26, 2023

Speakers:

Keywords:

curation

team

product

content

algorithms

editorial

moments

humans

work

twitter

curators

tech companies

google

platform

call

people

engineers

build

tweets

threads

Welcome to this morning's featured session. The show will begin in just a few moments. Please find your seat

Right Please take your seats our shows about to begin

Welcome to the humans who powered your algorithm. Please welcome to the stage Nikita Patel.

Hello, everybody, good morning. So, make some noise if you're staying at this hotel, and you've been up since the crack of dawn so this morning was quite entertaining but we are glad that you are here with us and I'm very excited to introduce this discussion today. So my name is Nikita Patel and I oversee leadership programs at Newmark J school at CUNY. I also currently serve on Oakland A's board of directors and I'm Secretary of the organization too. It brings me great joy to thank you thank you. brings me great joy to introduce this session because curation is very near and dear to my heart, having worked at Twitter myself and with some of our esteemed speakers as well. So without further ado, let me introduce the session. So the title is the humans who powered your algorithm the untold story of big tech curation, content and editorial teams. It's no secret that every platform has human curators, and as curators and journalists has led these teams, they've been far too quiet for too long. So today I'm excited to hear some real talk and some constructive feedback and solutions from these folks. They'll explain how they've kept watch over the algorithms, and they'll share valuable lessons about the integration of AI as well. Without further ado, I'd like to welcome our panelists to the stage. First, we have my former friend and colleague, or should I say formerly sorry. She is not a former friend. I was gonna make a joke about the platform formerly known as Twitter. My goodness, Joanna Geary. Welcome to the stage. Carolyn O'Hara from Pocket Mozilla Rachel Richardson from snap, Gigi Zang, formerly from Google, and our esteemed moderator David Smith Audra formerly from the platform known as Twitter. Right then, and now currently at politico, who is also RNase. President. All right. Once the conversation gets going, if you have questions for these folks, please use the conference app or the website to submit your questions. Thank you

right Good morning, thank you for coming. We're glad that everyone that all as well that all is okay. I also want to give a shout out to the OMA staff that once again proves it is ready for everything all of the unpredictable stuff that happens at our conference. And also just got the note that apparently both elevators are now open so we are back. Alright, I'm so excited. I think we're all excited to have this conversation today. As Nikita mentioned you know, this is actually a conversation that we've been having in back alleys and dark street corners for years. And it makes all the difference that we are now having this conversation on stage and the OMA stage in particular. So actually they kind of kick us off and give a sense of you know, how we started to connect the dots. I just wanted to go down the line and mentioned the first time that we each met, I guess one another and realize that there were humans doing this work at other other tech companies. So Carolyn, would you lead us off?

Absolutely. Thank you so much. Everyone, for coming today. So excited to have this conversation. Yeah, I think I had known that there were folks at other platforms I I'd met a couple in passing folks at Apple news or smart news up day. But it was earlier this year. Actually, when Joe and I got coffee and I just say it was kind of like a therapy session where we really started to kind of mind meld and talk about the challenges and opportunities and the cool work that our teams had done in the overlap between them. And yeah, so I don't know I credit my connection with Joe for really putting this together. And I've actually hired someone that she used to work with the past. So it's just been an incredible start to this cool web.

And for me, I knew Joanna from a former life we both worked at News Corp together and so when I started doing what I was doing at snap she had been at Twitter for a long time. And I reached out and we held hands ever since and she exchanged notes and swapped to war stories. And I think you actually do need a sort of an emotional support friend when in these roles and Joanna was mine.

Thank you. You remind you I was gonna say if you're talking about who I met, who ended up working in curation first, it was definitely Rachel in London Tiki Bar at a new school Christmas party, if I remember rightly. But in terms of finding someone just making that connection in this space it was so I've been in this space since 2015. It was 2018 and it was a OMA and it was David.

Ditto. Yeah, I met David almost seven years ago I Google actually when our team first started and you are the first curation person I met.

Yeah, and likewise because I'm so we thought yesterday as we were doing this, I thought it was Joanna in 2018. But it turned out it was chichi because you had you had curated a Flipboard earlier a couple years earlier than that. And as I was trying to build a curation team for a Google product, you know, I looked at Flipboard as like the shining beacon of like how it should be done in terms of like the co founders were both from tech and content and they had staff curation, and and finding someone who was on that team was like a massive discovery. So so we're gonna, we're gonna get into it. Emotional support has already been mentioned, but we do have like the level setting. We like the goal for this conversation is to open up and put on the record. I think things that we should have been talking about openly in this industry for a while. It will not be our therapy session that we may go there, here, here and there. But we really want to get to a place where we have, you know, can start a constructive conversation, especially as we're going in to this era of AI and like what are the lessons and the insights that we were able to pull from this work that can set us up for that? And so, just wanted to show kind of, again, that the example of how this work is really it's certainly I think swelled and maybe crusted but it's certainly not disappeared at all. Carolyn, you're still doing this work at pocket, but all of these companies have staffed curation teams at one point or another, even YouTube back in 2010, or excuse me before that 2006 2005 Just as they were getting started, that's how the YouTube homepage was populated was a team of curators that were identifying the best content that was on the platform, right so but we've never had this conversation together about about how we approach that work and what it entails. So let's get that going. Right now. And Carolyn, would you lead us off with the work you do at pocket?

Absolutely. Thank you. I think a pocket is actually like a pretty interesting product in this space because not only do we have humans doing curation, but pocket itself is very much a curation platform. We are millions of users have downloaded this product that allows you to save things to read later. They're essentially curating the internet for us. And that's a really powerful signal for us. And we use that data and those insights in terms of what people are saving, what they're engaging with anonymously and an aggregate to kind of get a signal on what's super interesting. And it I think the Save signal is actually particularly interesting because it is so distinct from the like, I think like can be very public and very performative and saving something is very kind of personal and means you might want to like come back and wrestle with it. So this the signal that we get is particularly interesting. I think we use that to inform all of the curation that we do. And so what you see on the screens here are a couple of manifestations of that. And it works differently with every with a whole we have a whole suite of editorial products, and it works differently with each one. Our pocket hits daily email, which some of you might get. It's a largely manually curated email that goes out to its everyone gets the same one each day, but it's informed by what our users are saving in other respects. Some of the other experiences that we see we're doing more the humans in the loop are informing the algorithm or improving the algorithm doing kind of content intelligence, and that manifests across a lot of different surfaces. We power recommendations into Firefox, not just in the US, but seven different markets around the world. So we're doing that not just in English, but in five different languages as well.

Great. And Joanna, talk us through moments and trends on on Twitter or formerly known as Twitter,

formerly known as Twitter. Yes, I will talk through moments and trends and all the work of curation, which before it was dramatically disbanded in November 2022 was a team of 130 people across 11 different time zones operating and curating in 10 different languages. And I couldn't be more proud of having the chance to work with those amazing individuals. Some of the more the point of curation was to bridge the gap between what people expected and wanted Twitter to do and what it actually could do. And ideally to use my the, the abilities of our team to then eventually be able to do that in an automated and scaled way. The visible stuff, the most visible stuff from our team is the stuff that you're seeing on the screens, you know, just trying to solve discovery problems with things like events and moments, context problems with what you can see very much Miss trend descriptions, and, and then also other things that maybe weren't quite as visible. So things like work helping to uplift credible information around like highly areas with high misinformation, topical categorization of tweets, and various other pieces of work that involved humans in the loop. So that was us, right?

We'll keep going through this bioswales keep calling out themes of you know, as an international operation, it was kind of built on a production desk method or model. And as you can see in the top commentary, screenshot in the middle there, you know, that was a feature that kind of blended the recommendations of the accounts that were relevant to a given story, but then the algorithm then surfaced the tweets from those accounts. So you started to get this blend of human and machine kind of working. compatibly so great. Rachel, take us through SNAP.

Sure. So for Snapchat, the editorial function which I managed, had a dual purpose, we were a walled garden that has actually changed a little bit since I left in 2021. But we essentially selected partners to be part of discover, as it was then known. And the other part of our role was also organizing content into topics which we called collections. So things like wild weather events, the Met Gala elections, we would essentially surface that content to the top of the feed, so that you know, something that we thought was a very important event for our users will be very visible because the rest of it was algorithmically sorted. And then we also had some ability to be able to control whether our users saw breaking news. So if something really dramatic had happened, like I don't know a former president getting indicted and having a mug shot we would be able to make sure that that piece of content that was very fresh and timely, was incredibly high on this essential feed, and sort of essentially counted against the algorithm that was optimized for engagement.

Excellent. Okay. And then you'll see two Google examples here because there are two kind of stages of evolution in our team at Google that chichi and I worked on. So the first one really emphasized our ability to structure the experience in the product around different stories and moments and so you can see the the 2018 midterm here, you can see the different tabs for latest stories, coverage of the Senate races, the House races, the individual states and then you go into those states and you can see the gubernatorial coverage and their specific Senate and House races. And so that structure and then the algorithms that kind of powered it. You know, our, our manager at the time, our engineering director, you know, MIT grad, serial startup entrepreneur, you know, engineering director at Google, he would go around and show this to his colleagues. And so there is no AI on the planet that could build this, like the idea that we could actually structure this experience and then have the algorithms populate it was kind of a breakthrough for us. And what we're particularly proud of is the you know, our team was was adamant that this is a US election. It is essential to have access to the information of civic value. So even though you can't look at a chart of traffic and say that on that basis, a Spanish language version is justifiable from pure traffic standpoint, we insisted on it and so our team put that together as well. And so we are proud of of that experience. But then Chi Chi, you were instrumental in kind of taking our team, I think, to like another stage of how we could work with engineering and improve the product to talk us through this.

Yeah, so I think that was closer to the halfway point. Before I parted ways with Google earlier this year. And that was in about 2019, where we evolved as a team from this role as curators to become more as consultants and closer to almost product managers in that regard. And this is a particular product project that I'm proud of where I worked very closely with engineers to design and surface in depth reporting from a variety of different sources. And the idea was to you know, highlight investigative reporting and in depth reporting beyond the headlines of what we were seeing in the Google News Feed. And so from the beginning ideation stage of working very closely with engineers, to determine what goes into the signals that we were using and the considerations that had to be made. And those were people, the people that I was working with, did not have an understanding of journalism or journalistic principles. So it was a really great partnership and being able to do that time and time again going forward. was how our team involved in adding more value to the product at scale.

And so if scale was on your bingo card that it just got hit, because I think that was something that came up for all of our teams, and I'm sure it comes up in your work. Carolyn is the ability to take this editorial knowledge and expertise and you know, inevitably, we would run into an engineer or an engineering leader in sec, but how does that scale? Right? And so I think that takes us to, you know, our challenges section which is, you know, these are great examples of what the work looks like now and some of our I think, kind of showcase moments, but we didn't get there overnight. So, to kind of open this up. This is a pretty big moment that happened shortly after a lot of our teams were formed. So in spring 2016, if you'll recall, there was a pretty big kerfuffle at Facebook. They had a trends curation team, and they there's a story that came out that someone on that team as an anonymous source, alleged that there is a process or a culture rather, of of kind of promoting liberal content and and down ranking or shadow banning conservative content. Right. And so that, you know, I refer to as the Spectre that I think loomed over a lot of our work. And also, you can kind of see this on the right you know, that mindset I think is still there. Apologize for the small point size but you know, Alex Heath on threads a few weeks ago just asked you know, openly How will threads, you know, handle news and Adam mosseri from meta essentially said it won't. The goal is not to replace Twitter. We want to have all communities here, right? And so we don't want to focus just on news we wanted to all sorts of community content. And fair enough. But I don't know that that's a decision that a platform is in control of right. It's the audience that that decides how to use it. And so I'd like to get to this question of kind of how our teams were formed and what we understood the charter to be when those teams were formed. So instead of going down the line, I'll just kind of open it up. And I'm curious for Joanna, it's your show. I

mean, it was wild. It was wild. There's some people in the audience you were there a little bit before I was like, but he might correct me later, but I think like we were brought in. I think, you know, I talked earlier about curation existed because there was an expectation of what technology could do on Twitter, and sometimes that couldn't be done and so humans would bridge the gap. And that's exactly why we were bought in we were bought in because there was a discovery problem on Twitter and a desire for people to want to follow along to be with big events live. And so a number of engineers got together and came up with what seemed like a really, really cool, automated solution to that during a hack week, and it got sponsored by the CEO and got built out and I think somewhere along the line of building out they realized they couldn't actually do that in a fully automated way, and that it was going to need humans, which needed humans in a wild amount of ways. So not only did it need humans to curate these things for moments and pull them together, but also the live discovery required required humans to sit there and watch a live feed of tweet and select them in real time to push them to you. So like I remember a lot of like, times, just like during the Olympics, just sitting on my sofa, just pushing tweets to people. So it was very, very manual at the beginning. Over time, we did go on the same journey that Gigi was was talking about around towards automation and using intelligence to actually provide humans in the loop. But it started there. And that became a theme. I think we continued to bridge those gaps.

Rachel, I'm curious that snap you mentioned it was you know, as a walled garden and as a partner driven experience, can you kind of talk through how your team was the remit your team was given to make sure that that was a strong experience?

Sure, yeah. We were very much sort of judged on what we deemed internally as like editorial Excellence, which was essentially like selecting the right partners that we'll be able to produce for the platform and do a really good job helping those partners sharing with them best practices, platform trends, that kind of thing. And then essentially, I think the key part of our role, which sits in the curation part was almost like fighting the algorithm, which as I said earlier, was optimized for engagement, which meant that it didn't always give the best experience at the right time, if that makes sense. And so, you know, we were sort of empowered with similarly Can I swear chef's tools? I mean, yeah, we were given some really chunky bits of equipment and and software and tools to essentially try and make sure that it moments particularly around news, or particularly around, things like the Super Bowl, or the Met Gala, or the Olympics is actually a really good example, to make sure that things felt fresh. And I think that's always probably still to this day for my experience as a user at snap is it's never quite figured out the fresh part of making that feed feel really fresh and new and relevant and recent. And so that was where we really intervened with really Shanky tools.

i It resonates so much this like phrase bridging the gap, because I think that that is this common misconception that I had before going that the algorithms were going to be so good that they were going to be able to provide this really fantastic experience. The algorithms are good, they're getting better all the time. But it's the work of teams like ours who are improving them and influencing them and bridging the gap where they aren't producing the right kind of experience. To meet your company's mission and values. And I feel really fortunate I pocket as part of Mozilla like we are not optimally optimized for engagement that is not like we do not chase clicks. We actually want to provide a really cool, balanced, interesting content environment and that feels great what I what I also hear as you all talk is like, it kind of starts like the the seed of the team starts in this like bridging the gap, we need to provide this cool experience to users. And then over time, you start to wear so many hats, you're just covered in hats, and it becomes you're you're you're helping build tools, you're working with engineering teams to build improved tools you are on the frontlines of better understanding and getting user insights not just from your own market but from markets around the world and bringing those back to the product teams to provide a an international perspective for engagement and what's interesting, your partner you're the bridge from your company to the media industry, because you've often most of your team has worked in newsrooms, and folks in tech companies often haven't. And you can be a really great advocate internally for understanding the challenges in the journalism industry. And and feed back to tech companies, what our pain points in the industry and like how you might be able to develop products and solutions for the media industry that would be get better content in front of more people.

Just wanted to jump in and like like underline that like I agree like there was so many times where our team was able to stop a predominantly US centric engineering team from doing something that would have really upset people in other countries around the world. It was and I think that's like from talking to you guys often. Like the curation teams are the most international teams within product if they are lucky enough to sit in products, and I think that is such a superpower for organizations that have them because there is a lot of clouded thinking around how to serve international audiences often when everything so centralized in the US.

Yeah, I just wanted me I wanted to add that I think it's fair to say from the very beginning our team was an experiment and an experiment meaning that we weren't full time employees of Google we were contractors, meaning that we didn't have job security. So at any given time, you know, your contract can be terminated. I mean, luckily, the experiment was a success. But the backdrop of that was two very difficult challenges that we had to overcome. One was organizational, and the other was cultural. And anytime you're trying to run an experiment that is not necessarily against but so vastly different from the culture and the organizational setup of a company. It's going to be really hard and we had a lot of growing pains, especially in the early days of that everything all of you said with the evolution resonated with me, but it required a lot of learning and, and work on our team's part to I think get to where we eventually ended up but trying to change a culture of you know, 200,000 employees. That is very, very interesting. And an organization where our team actually sat with engineering, but we weren't engineers. And we were close to closer to product management. But we didn't sit with product product managers either. So when we were working or trying to work with, you know, partners on UX and product and engine and they didn't know how to work with us, and it was oftentimes a struggle.

Yeah, I couldn't agree more with that kind of description. And I think it resonates also with with remarks earlier that our teams often were kind of identified as the team that could do so much of this work that was from a tech perspective or a tech company perspective. Undefined, right. So tech companies have, they have a pm job ladder they have a UX designer job ladder, they have an engineer job ladder, there was no curator, editorial specialist job ladder, except I think at Twitter, we had the curator ladder.

But then she eventually spent a long time working.

And even when I was trying to hire curators at Twitter, I'd worked with a recruiter and it's a What do you mean curator job ladder, so it wasn't as ingrained. But, I mean, that was I mean, to teach his point, like, it was, I think these things go together, where the our work and our expertise was not understood. So it was it was staffed and drawn upon from a bag of money to hire contractors, as opposed to a defined function in the company and in the in the industry. I think we all kind of have lived through that.

Yeah, I would say one of my proudest anyone who's in management out there, you know that some of your proudest moments no one else gives a damn about right like, no one really cares. But one of the things I really cared about was working to convert our team from contractors to full time staff and to bring that sense that we were actually a team of specialists that you needed in a company like Twitter. But it was really hard. I remember, you know, one person in my team coming to me and say, Why is it I'm the only one in a meeting that has to introduce my name, and my team, and then what that team does, and I have to give a full introductory presentation every single time and it hasn't stopped and I've been here years. And it's just because, you know, teams move around. They change and people didn't have experience of our team and what it did, and that was always a problem and made things harder. I mean, that culture change harder. I think another thing that made the culture change harder is and I think again, people in news and doesn't understand this content. Has this terrible Dunning Kruger effect because people are swimming in content all the time. They think they understand how it gets to them, how it's made the thought the thought behind it and that is exactly true of everyone who works in tech companies as well. And so they weren't really aware of what they didn't know. And a good really good example of this is I think it was maybe 48 hours before the launch of moments in 2015. There was an edict that came down from Jack Dorsey who said, it's going to be a cleaner product, if all your headlines only have three words in them. And so if you ever remember the launch of moments, we did indeed launch with only three words in the headlines. Which somewhat undermined the discovery capabilities of the product for about maybe a week, until it was reversed. And again, that was, you know, it was a very well meaning person who understood product and understand that lots of words looks messy and you want to streamline your product and create that whitespace and he understood design. He just didn't understand content. And you have to take the entire company on that journey. And that's hard, especially if you're predominantly contractors.

I think Can I just chime in here as well? I think the the whole contractor sort of full time employee thing is really fascinating and I had a lot of contractors in my team, which I felt very uncomfortable about. But I do think at the heart of the issue is this very strange thing with tech companies. And I think particularly going back to this issue of the specter where they never wanted to admit that they had journalists in the building, and they never wants to admit that there was an editorial team because that meant there were people with levers, and people doing things in some mysterious room somewhere. And that was always to me just so odd. And I understand the politics and the columns of it all. But it was just so bizarre that they like to point to like, oh, but we do have journalists that work in the team. But we were never really given that much of an opportunity to explain what we did or what the work was because I think it was always really, there was always a really big concern that it would be misconstrued. Because I think as soon as you start saying, Oh, yes, we do curate things into topics then you do get you know, the sort of theories from different sides of the political spectrum, that you're actually trying to manipulate the audience in some way. And, you know, I don't think that's what my team did at all, in any of the powers that we had to you know, elevate content. But that was always always the concern.

Yeah, there's nothing more mysterious than an engineer working on an algorithm on their own time and coming up and saying, Hey, I've discovered breaking news. We can just use this on the front end. Right. And but, you know, that's not how people think. And so there was often way more oversight. On our teams than there were on anyone building discovery algorithms on the platform laying. Yes, Discovery algorithms with all would go through a test and approve process but they would often be tested out in the wild.

Yeah, I think this distinction that has emerged now it has become clear between trust and safety work, which is identifying the toxic stuff, the unhealthy stuff and actioning on it appropriately, and curation work or editorial work. I mean, it certainly wasn't clear in my mind in 2015 2016, because, you know, we were managing teams that were seeing all of it, and so we often came across so much of that content. But I don't think it was clear to the people around us either, like the charters that we had as teams. And so I think that's, that's been one distinction. That's that's surfaced over time. But it is it's about finding that common language and understanding where the gaps you know, as Carolyn is you're mentioning, where those gaps are and what are the strengths of each of each team to fill those gaps. So I want to get around as well to you know, we've talked about TVCs or types of vendors, contractors, and how we had, you know, that's how we started, as I talked about job titles, and what we call this work Carolyn, since you're still doing this work, and you have a team start us off, but maybe kind of has there been some evolution there in terms of what you've called your team or what you've been asked to call your team,

so much evolution and we are nothing, nothing, if not wordsmiths, right. So yeah, and to your point earlier there, there aren't defined. I mean, there aren't defined ladders at these companies and is still a work in progress. Like when you go to open up new roles. It is a bit of a circuitous journey, and occasion and it's not something that HR is immediately like, Oh, yes, I understand this profile. And so our team, we sit on the content team, that has been called a number of different things over the years and if you go to different people in the company today, I'm sure they would call us the curation team, or the editorial team or the content discovery team. So it's all a bit of a journey. And ultimately, I think that's a big part of what I do in many respects is advocating for the work that we do, showcasing our successes and how we can plug in and how we can improve these products constantly. I I don't try to get too hung up on what we're called, though, you know, it's it's it's easier sometimes when you can get some things distinct so that you can get time with product and engineering teams. But I think the you know, our path to success has been showing our successes showing how we can create better products for end users. And that's ultimately what I think we do. But we're, we're called content now, which define as you like,

Rachel? Yeah, I was I was the head of editorial and an editorial team that was part of a content and partnerships team. So we were pretty straight shooting I'd say. And from the get go, we were we were like that we definitely had some issues with ladders. That actually I'd forgotten about that. And now that it's all come flooding. Sorry about that. I know. We did get there in the end. But it was always incredibly difficult to try and figure out what a career path look like for someone in content because it really did stop somewhere midway down the sort of overall chart because you know, that person was never going to become the VP of engineering.

So we started out being editors and edit tutorial for a couple of months until our legal team found out that's what we were calling ourselves and then asked us not to. And then we became curation and curators after that so it got a bit messy for a while. And curator we needed to work hard to get everyone to really accept the word curator. A lot of people who are coming in for news were like, that is not what I am. I am editorial, but I'm glad we did not least because a couple of years later, I discovered that there were a lot of very high level executives that don't know the difference between editorial and editorializing in word definitions, and so editorial to then kind of reinforce the spooky cabal of people who had opinions that they were placing out into the wild. So curation was really great for us because it gave us like this, this slight gap. It wasn't that I think we were in discussion about changing the team's name. No one knew that I think at the time but into content intelligence because ultimately like going back and reading that's quite sexy. Yes. Yeah, it

sounded content intelligent intelligent. So

in full disclosure, I stole that from Netflix because they have they call their function content intelligence. Yeah.

So we were gonna steal that from Netflix. Thanks, Netflix.

The same answer, actually, as job because you talked about the editorial position, and any whiff of that at Google, I think we would have preferred to call ourselves editorial that would have been just right on what we were doing, but that term just was people were not having it. So there were a couple of options obviously curation content strategy and ultimately landed on product experience, because we wanted our job was to build a great product experience for our users, but also our publishers. And you know, that's what we landed on. But I think the hard part is that it actually doesn't mean anything to anyone at Google. So it still required a lot of explanation of, you know, what is it that we do what our skills and

and some of the backstory here that so at the time, you know, we were going from being the editorial team for a previous product called Google Play Newsstand. We were reordered into Google News to help become the new Google News. Google News is a search property. So all the principles and the constraints are different. So we could not call ourselves editorial anymore. And it was about a six month endeavor saga journey of trying to figure out what to call ourselves and I will forever feel terrible for chichi and the rest of the team that had to like endure that because I had no answer for them. I couldn't get an approved answer for our team. And at one point, I flew out to King's Landing or Mountain View and got time with like the engineer VP, who would who would be the decider on this, and I put a slide deck up and I proposed editorial strategist. And the whole meeting took a left turn, got we kind of went back and forth. And then you know, as we're closing the meeting, you know, just one of those things you say is like, okay, you know, thanks. I'll send you the slides and we can follow up. He says, No, David, do not send me these slides. I don't want these in my inbox. I want you to print this out and burn it. And that goes back to I mean, this was within a year of what you see on the screen like the anxiety was so high around at a company level, the perception, nevermind the reality, the nuance of the work, we did the perception of having editorially minded people being used as a cudgel against us, right. And so even having it in our Google Drive was seen as a liability

my life would have been so much easier if I was allowed to be influencing the content in like manipulative and like malicious ways. Like I would have been a lot easier like the job was so locked down and you were under such scrutiny internally because of that fear. That's when people turn around, say, Oh, you were just there, deciding whatever content you wanted to put up, and it was great. You know, it just makes me laugh because the stress of the job came from that oversight.

Great. So I'm going to do a plug here as we're kind of approaching the final third. If you have questions and you want to put them into the app. We'll have an Akita reviewing those and kind of sending those to us at the end. But let's get to some of the lessons and the takeaways from from doing this work. And I wanted to show here kind of a you can see a tweet from Russia. Right after the events of early November in which the curation team along with 3500 colleagues at Twitter were laid off. And you notice 18,000 likes, right? I don't think that that recognition of curations work and value on the platform. It certainly wasn't there on day one. It wasn't there a couple of years. In but by the end it's certainly was right. And then if you look at kind of a thread I captured from from Ezra Klein, who many of you may know has been covering AI and I think more broadly kind of curation and content for a couple years now with with pretty intense focus just expresses this desire for the curatorial internet versus the algorithmic internets. I thought that really nicely captured. You know what the audience is expecting what the audience wants, and there's validation that it can't just be about pushing down the bad stuff. It's got to be about contextualizing and promoting and pushing forward. The best stuff, right and finding a way to do that. So for the second, I'd like to ask kind of what was what was an aha moment? Maybe where your team finally hit paydirt or struck gold and you found that engineering colleague or that PM, who you really could have been a product launch or it could have been an insight where you started to speak a common language. Can any of you think of a moment that that comes to mind

2020. And the pandemic started, there was a summer of social justice protests and it was just this, this this, this moment, when people craved high quality information with context from people they felt like they could trust and you didn't in all of a sudden were like, who is the pandemic expert and who is the underserved voice that needs to be elevated because they can actually speak to this really trying moment. And the content team was the one that we kind of pioneered these rapid collections where we would go out and find experts and elevate them and push them through our recommendations. To help them to get their human voices into into our recommendations. And that was when I think the whole company was like, that's a value add. Like that helps us and Mozilla is distinct too because we have a manifesto we have a mission like we we we have ideas about what the healthier internet looks like and what healthy content on the internet looks like. And that was able to to push forward our mission. And that was when we started to get tooling resources and the rest and you know, that was a really great product we ended up continue with that now has become like a flagship editorial product for us, where we work with individual experts, subject matter experts and curators and make these really fantastic collections and they are a huge part of what we do today. So that really felt like at that moment, that moment in time that came out of need really unlocked that for us.

Yeah, I would agree actually never has a Something moved up the priority list, like the pandemic like products that I was asking for for years was suddenly available within like a few days. And I was like, Aha, you can do it. It just took a global it took a global pandemic. But yeah, that it was really interesting in as much as that was positive. It did also feel quite negative because it was like Oh, right. You just like didn't believe me before. Like, this need was not like I hadn't communicated the need enough or you hadn't understood the need enough. But yeah, there was there was a primal fear within not only snap, I think it was every company, right that they would be responsible for spreading misinformation, and that would actually result in people dying. And so yeah, the idea that we could create collections around, you know, life saving health information for young people was was really important and it galvanized you know, lots of different parts of the company to come together to produce ways for us to curate, and everyone was so comfortable with that in a way that, you know, there had been some expressions of discomfort around before whether like, oh, yeah, health information. Yeah, that's great. And obviously the pandemic became an incredibly divisive subject as time went on. But if you go back to March 2020, there was a sense of unity that I felt anyway, that we were like, oh, yeah, we should actually have information from the World Health Organization. It's going to help people see pandemic for me too.

I think real change takes a long time. And I can't say that's, you know, completely happen. Like Google, but I think we made a lot of headway. And there were many aha moments, I think, throughout my time there but it was a lot of the change that happened happened over time. It wasn't necessarily I think, one moment in particular, and our team I think what we found so starting off in a place that was frustrating, as contractors and challenging, I think we quickly learned that we need to learn what the interests and the language of our key stakeholders and who they are within the organization, if we actually want to get work done. Right. And so we did that work and in addition to that, we also brought on journalists that we, you know, love and we're very strong voices like Maria Maria ReSSA, came to speak. And we had a monthly speaker series that we did, and over time, especially toward the end You know, this was something that was reflected the change was something that was reflected in not only the people that we hired, that came from newsrooms, whether they were UX or product managers or engineers, but also in the type of ideas that were seated and spread throughout the organization. So for example, after talks, it would really get, you know, people inspired about, well, they talked about this problem, how what can we do as Google to help them and change them because they really did care. But I definitely think that took time.

I would, I would echo the idea that it takes time. And I think I'll kind of briefly talk about two are home moments. One was something that took a long time, and that was having, having this team inside of Twitter and thinking that my focus was to try and convince everybody at the company about the value of this team, and to talk to them about it. I realized suddenly that actually, our role was to understand them and their language, and we've been trying to convince them of our value. Using our language. And so we ended up taking on some pretty meaty projects, teaching the team product development principles, and also data literacy. And we ran quite a lot of work to get people to understand what it takes to take two instead of just having an idea having a hypothesis and going and testing it and then getting buy in for it and so on and so on as per tech companies that really helped to change once we changed our language that helped how we worked with people. And then I think it like a very recent aha moment was again something that was long time in the making. But as anyone who works in change knows when something good happens, you grab it, and then you tell everyone about it as much as possible to try and make more of it happen. And we had for a long time known that placing humans in the loop of certain types of algorithms would we believed strengthen those algorithms and and improve the output, but we weren't necessarily getting buy in directly from the machine learning teams. And so we did put a huge amount of effort David came on board as well to try and help to build the case for that. And just and one of the areas where we think that's really important is in the area where you have to over index on something called precision, which is getting the right stuff out to the algorithm and recall, which is getting stuff out to the algorithm. So when it's high precision, we could offer a benefit. And we managed to do a piece of work with our trust. And safety teams around out of network push notifications. And we were looking to reduce the amount of tweets with misinformation that were being pushed to people who didn't ask about them. And we managed to do that with content intelligence in a way that reduced in the US and Brazil reduce those that happening by millions and millions and millions of impressions on tweets every month. And that was such an exciting moment for us. We were preparing to launch that worldwide. And we had seen that as something as a way to prove out humans in the loop are really valuable and it would have been you'll have to believe me because we don't exist anymore.

So we'll turn to questions in a quick second I want to kind of slide in the my own aha moment, which was I discovered that the the key to making a project happen for most of my time at Google running this team was finding the mid level engineering manager who had written lots of code manage other engineers probably came from a startup so had a very kind of rogue sense of rules that they could operate by, and was not beholden to like the company level. party line on a lot of these things. In other words, they just like they saw the gaps, as we've talked about, and how we could fill them and they would just code up a tool to help us do that. And then seeing that result and you just you hold it up to the proximate like this is so much better. Right. So we'll turn to questions I'll leave these up while we have questions if you want to keep you know studying this area. These are just kind of three directions you can go if you haven't listened to me ACWA real close Flipboard podcast on the art of curation, I cannot recommend it enough. It is just an exceptional body of work and she's done a huge service interviewing curators from across multiple fields and domains. Michael Bhaskara is book is kind of the definitive book and then Twitter had its own curation guidelines that are still on the site, please visit them if you if you so choose. I cannot instruct you but if you would like to save them while they're still there, you may do that. Because I think they are kind of a seminal series of documents for this work. So with that, Nikita, would you kick us off with questions?

Sure. We have many questions. Many, many actually. I wish we could get to them all but the first one I'd like to ask is from Tom from fathom. So what does the decline or end of curation teams mean for the 2024 election? And what does it mean for the rest of the journalism industry that still relies on platforms for distribution?

No, I don't I feel like it's a Twitter question. Well I can't take it though. If you want me.

Well, let me let me say this first. Curation teams haven't declined. The reason why there's a lot of form of people on here is it's much easier for us to speak when we don't work for the companies anymore, but they are still there. Minus Twitter. Twitter is another question and I cannot speak to the current ownership strategy plans vision for how this election is going to go. I wish them well. But curation is still a massive part of many, many platforms and increasingly so and I think what actually you're going to see is that newsrooms are going to end up working more closely with some of these teams, and also taking on curatorial roles that look very and feel very similar themselves as different platforms start to become important to news.

The only thing I'd add there is that I think of all the major platforms Twitter is going to be the outlier. But I do think threads is really fascinating in how they are adopting this sort of no news approach. And one example that happened this week was something that Ron DeSantis said in the GOP debate, and I cannot quote it, but along the lines of sort of killing dead, drug dealers crossing the border, which was very threatening the language that was used that that was take anyone who posts that on threads had it removed. And so it is going to be a really fascinating and like as an armchair observer, I find it really fascinating to see what happens to threads in this next year. But from a Snapchat perspective, there's still a lot of guardrails in place. I think from a Google perspective, I'm assuming you guys feel pretty confident there. And I do want to

add, I mean, agreed on all these points. It's, it's absolutely not just a US problem. I mean, it's it's 1,000%. It's a global problem. I read a stat there 3 billion people voting next year. The end, you can be absolutely sure that that is a stat that is known at Tech platforms that is huge reputational product risk brand risk. They do not want to be hostile in front of congressional committees. They are understanding the need and the problem and their I believe tackling it and it's going to require teams like ours to help them tackle it.

I did see Twitter is currently hiring for election threats analyst if anyone is interested. I mean X not Twitter is hiring.

All right, thank you all. So next question. As a social media editor who used to pitch Twitter moments, it was always seemed like some outlets were favored and getting featured more than others. And I've always wondered if these places had deals with Twitter. Was that the case? I'm actually going to take this before I hand off because that was my job. I was head of news partnerships for Twitter and I will say that there were no deals. You know, we just had an incredible amount of publishers, not only here in the US, but globally right publishing moment. Some publishes multiple moments on a daily basis like it was. It was really quite incredible inspiring, and probably a handful for the curation team to manage and so the factors from a partnerships perspective, whether freshness of moments, the relevance of moments, where they're not duplicative of the curation team's efforts, were they complementary, right? Did they add more context and were they rich media and things like that? There were many, many factors that I would train my partners in terms of curating, and creating and putting moments and I'd love to turn it over to you all now.

Yeah, for Google Play Newsstand. There was a brief while that we were receiving pitches via slack, which by the way, the Apple news team does as well. And many of you may participate in that but there were no deals from our side. And mysterious about SNAP because snap was the model. It was a partnership. Yeah, I

mean, we did have deals. Yeah, I can't speak to how it is now. But back in 2021, it was a walled garden. You know, partners were sort of accepted on a basis of editorial, editorial quality and balance. So there would definitely be issues where we were, you know, a really fantastic partner, but we'd be like, Oh, we've actually got loads of this and we actually need more of the other thing.

Yeah, my hunch Carolyn with pockets since it's so user signal driven that there's not a deal elements to partners, there's no

deal element. Now we do have a syndication program wherein we license older, more evergreen content as a way to extend the long tail on really great content, and really just to build bridges with the media industry and elevate great content on the web. But no, we don't take pitches. We don't have deals.

I think, good guy, Jenna.

So Nikita mostly answered the question, but just as a little bit of a context, Twitter pivoted away from the moments products pretty quickly. They decided like they decided the tool for discovery was really important, but they had to kind of over branded it. The CEO was leaving and he was his like, big last hurrah. And I think they had over branded it as a product rather than a feature. So there was a lot of engineering and results was spun out pretty quickly from that team, which meant that every submission from a news provider had to be manually placed into a certain position and various things done in some difficult tools to get them to be featured. So we tried to feature as many as we could but it was a real pain in the ass. So sorry. We did We did our best to Todd. All right. We

have time for one more quick question and quick answer. Go ahead.

Okay, very good. So this one is from Andrew Fitzgerald. Who is

he who is he's a plant.

For those who don't know Andrew is the founder of the curation team at Twitter. Exactly. The father of curation at George Washington of curation.

The question is, it seems inevitable that machines are a long way from understanding human content. Do you think the pendulum swings back for the content based tech companies like Google and meta? Or do you think that engineers have finally won this battle?

Can I take that one? I I spend a lot of my time telling people not to be scared of AI. And I don't mean that that completely literally. I think we should be scared in the future. But I think it is completely overstated how powerful any of these things are right now, because I have seen how bad they are. And I have worked with machine learning teams. And I think if everyone could spend some time with an engineering team who are trying to create some sort of machine learning algorithm that in any way reflects human interest, like figuring out what funny looks like it is a hot mess, express of disaster. And so I think in the future, like look, maybe we're you know, it all ends incredibly badly, but as we sit here right now, I don't believe any of these things are that powerful to replace humans? That doesn't get away from the very sticky situation of the where the humans fit with the algorithms. I don't have any answers for that either. But I feel quite strongly that it's never as good as they say it is.

Can I just really say we've got to stop the pendulum from frickin swinging. Like if the pendulum stopped swinging between humans and algorithms and humans in our hands, and we actually worked on stuff from the beginning. blending the two, it would work really well. We know it could work really well. So I'm just like please someone stop that frickin swinging.

Yeah, I don't think it's it's not a win lose. It's it's what it what are the humans do best what are the contextualization packaging like content understanding that the editorial brain that you can still bring to to add serendipity and delight into these products is something that the machines can't do? Machines can help a lot. They can make it way better, they can scale it, they can make it personalization, they can understand the data and it's just so it's it shouldn't be an either or I am so really compelled. Like, I think everyone's shared that as reclined tweet, I mean, it made the rounds and all of its lacks. The character like the curatorial internet is really interesting. People want that. And I think we have to like look at what consumers are after and lean into that.

It is the coolest job in the world. When it works and you're collaborating with these people. It is the coolest job in the world. And I feel so privileged to have had a chance to do it.

100% So Andrew, we reject the premise of your question. And and it is a super cool body of work that we are excited to finally talk about and that we are glad this conversation is out in the open and will keep continuing as a closing note. I didn't have time to put it into the slides, but I threaded at can't be the right language. Casey newton of platformer who's you know, reporting many of you know has covered this space for so long. And I asked him what he would ask us if he were in the room today. And he simply responded, I think I would just say thank you for trying. So we'll leave it there. Thank you so much to all of you for attending. Thank you to our panel. Thank you to RNA for putting this conversation. On. The program.