This podcast is brought to you by the Albany public library main branch and the generosity of listeners like you. What is a podcast? God daddy these, people talk as much as you do! Razib Khan's Unsupervised Learning.
Hey, everybody. So another episode of the Unsupervised Learning Podcast. And I'm here with my old friend, William Gunn. William and I have known each other for almost 20 years now, at least when it comes to the internet, which is a little different than the real life. But still, you know, this is the 21st century. William, can you introduce yourself to the audience?
Well, hey, everyone, I'm William Gunn. I'm the Head of Communications at Quora. And, yes, you know, a very long friend of Razibs. But still, like, it's probably the oldest acquaintance that I've never met in person.
Well, you live in the Bay Area, right?
That's right.
All right. So we'll make it happen after, after delta is over, because I do go, you know, to the Bay Area fairly frequently. And so we'll have a beer for old times sake. But, you know, speaking of old times, so I knew I first got to know you, I think you were at Tulane, you were doing was a biochemistry?
molecular biology, yeah
molecular biology. So you're a molecular biologist by training. And the next thing I know you're working for Mendeley, and then you went to Elsevier, and now you're Quora. So you've been in this like, quote, "science communication game", in a way, professionally, for a while now. And I know, it's been like, you know, you know, working at Elsevier, let's just say that you were at the center of a lot of drama. And, you know, that's, you know, they paid you like, that's, that's the game you chose. So, you know, I don't feel sorry for you. But I know that it was kind of, you know, kind of a difficult balancing act, I feel because you are a liaison to the scientific community. And scientists have their own opinions. And they have opinions about corporations. And then you have the corporation you work for, and you got to be respectful and loyal to them. They're paying your salary. I mean, they're paying you for a job. So can you talk about like this, this, you know, kind of peregrination that you've had over the last like, say, 15 years or so,
You know, if I had to go back and give a talk to people about how I ended up doing what I'm doing. It would not at all, follow any of the traditional career advice or anything, you know, so I thought I was going to be a scientist, I was going to be in a lab, you know, doing research. A lot of my colleagues are still in the lab doing research. But life sometimes just throws you curveballs. And it's how you how you ride them out. Right? Do you just a wave, you know, knock you over? Or do you surf it? In my particular case, it was to Hurricane Katrina, which came through New Orleans. When I was finishing my PhD
Yeah, 2005 for the Zoomers out there, because they probably know a woman, they probably know a girl named Katrina, but they don't know, the hurricane. Go on
So yeah. Gosh, it's dating myself here. But yeah. So I was finishing my PhD. We had to evacuate New Orleans, I ended up working in a collaborators lab at the NIH for a little while. And then our whole graduate program, or the lab was in up and moved to Texas while everyone was still kind of in this, you know, Hurricane diaspora, and that left me sort of like without a lab, so I was just writing my PhD, I didn't was done with all my work, but I didn't have a didn't have a lab and there's so there's really no reason for me to, to be there. And so I thought, well, you know, why don't I start looking for, you know, what I want to do after this. And you know, there was I was looking both directions, like there was, there was some opportunities for me to do some postdocs. But then I got this offer - this offer like to come out for the summer and do a short term consulting gig for this startup company, which I can now describe to people as, as a like Theranos but real. And I just did that for the summer thinking, you know, like, I don't have to be in New Orleans. So right - my, you know, my dissertation that can be you know, working as a consultant and writing it up. Now, of course, anyone who's tried to write a dissertation and have a full time job at the same time can probably guess where that where that led. But it got me interested in the industry side of things. And so then, when Mendeley sort of came along, they actually reached out in this life science friendfeed group for...
Well most, most of the scientists will know what Mendeley is. But can you tell what Mendeley is to the non scientists in the audience?
Yeah, so Mendeley is a tool for researchers to organize and share and collaborate around their publications and other people's publications. So let's say you have, let's say you've - you're writing about stem cells, and you have published a few papers. And you are trying to follow this particular niche that you're in. So you can go to PubMed, and you can do searches, and try to keep on top of the literature that way, but it's, it's really, really hard, right. And so, back at the time, we kind of thought, you know, there's like this thing called last.fm. And there's, you know, Spotify wasn't quite around yet. But basically, this concept of collaborative filtering, filtering, where we thought, hey, you know, if I'm reading, like these papers, we should be able to have a system to say, like, other people who are reading these papers, also read this other paper, maybe I should read it too. And so we, we kind of put together the system to, to do that. And then we realized, oh, you know, it's kind of handy for everyone to share, like a citation library. So if you're working on a paper with someone, then you can automatically generate, you know, a reference list. And your collaborators can do the same thing, because you're sharing this reference library. And they the UX of the existing reference managers were just terrible. So it was like, you know, a fish in the water to move into that space. And then, you know, then, like, there's all kinds of data products and things that came out of having a sense of who's reading what, and kind of like, how the attention to the literature is being allocated upstream of citations. And so that was kind of, you know, a fascinating thing there, which got me a little bit interested in sort of the whole meta science angle of things. And then, you know, after a few years, like you said, Elsevier came along, they acquired the - they acquired Mendeley. And I stayed with them for a little while. And then I moved on to working in a communications function at Elsevier more generally, because I had done a lot of the original community development of - at Mendeley. And so it made sense for me to, to stay in sort of that communications, community role. And what I thought I might be doing at Elsevier, is helping the product designers at Elsevier, you know, get a clue about what the research - researchers think, because a lot a lot of people don't know about Elsevier is they're not just a publishing organization, you know, they have all these information products, for doctors and for nurses and for, like drug researchers. And also for people who, like librarians and research admin, they want to know, like, if we recruit, you know, Dr. Big Wik, so and so is he going to be worth the salary? Right? And that kind of thing. So this, so that was valuable from a data point of view. And I thought it would be really useful, like, okay, so we can bring in Mendeley we can bring in this new real time data, you know, and maybe we can give them a competitor to like, the impact factor, which you know, is from Thomson Reuters, a big Elsevier competitor. But, you know, it turns out and I can, you know, I can say this now, that Elsevier really has a hard time... I'm choosing my words diplomatically. It has a hard time with a modern product development paradigm, especially like a user centric one. And they, I think, they just kind of really couldn't get it together. And also like, there's all this internal politics you face when, like, anything that you might want to do. That like affects, you know, the publication business in some way. Like that's, that's just not going to happen. And then and so it was a
it was my my big hopes, my you know that going to Elsevier, we're going to be able to like, you know, modernize the way they do it and really bring out some new things and get rid of the impact factor and bring in you know, this this great stuff it just, it's just too slow and conservative and it couldn't, couldn't get it out. That said like, the whole institutional sales piece was a big part of it too. Because, you know, if if anyone out there is like thinking of building a, you know, an app for scientist to do something. It's, you know, look me up, come talk to me, I can tell you like, how did not spend years trying to to do institutional deals and ending up with nothing. So, yeah, I mean, that's kind of that was through Elsevier.
Yeah. And so I mean, again, like the the scientists in the audience, they'll probably all the scholars will know what Elsevier is, but or was because it's got a new name, and it's merged with something else. But I think RELX or something RELX. Yeah, all done. So Elsevier or like RELX. Its revenue is £7 billion pounds. Its income, operating income. So like, you know, I think that means like, you know, minus the cost is £1.5 billion. It's got, like, how many employees it's got a lot of employees got 33,000 employees. Yeah. And so like, you're so I mean, and you know, the stuff that you're telling me one, it's not surprising, because I, you know, whatever, like, You're old enough, you've seen things, but also, you have a big social media presence on Twitter. And I could tell that you kind of the whole time you were there, a lot of the times you were there, you're kind of on a knife's edge of just like, Okay, you can't say anything bad about your employer, but you can see where people are coming from with some criticism, is that correct?
You could totally see where people coming coming from with a lot of the criticisms, I also got to see, and this was, you know, it was a very eye opening experience. And, you know, I'm under no, you know, legal or financial obligation to say anything nice to them at this moment. But there are a lot of complaints and criticisms that come from people in the academic world, that are just totally out of touch with with reality. And that kind of fall in the category of like, you know, a, what we would call bikeshedding, you know, just making a huge, huge, huge deal out of something that's really just minor and not actually a big deal. So, I mean, having having that the, the perspective of seeing like, here, here are the actual real problems, because I've been on all sides, I've been on the researcher side, I've been on like the, you know, the service provider side, I've been on the, you know, the publisher side, and seeing all all these pieces together and seeing like, and then, you know, frequently interacting with funders, you get a sense of like, this is, these are the things that really are screwed up about the research process. Overall, this is what I wish people would, you know, complain and mobilize about. And here are all the things that people actually complain about.
Yeah, well, so can you can you talk about, um, you know, can you talk about, like something that people complained about? And then like, really, they're just complaining to complain, and maybe get a few likes getting a little clout on social media, but there's no follow through or anything - Because I know that it is, it is easy to complain about the private sector, and what academics called industry, but then actually, following through and providing an alternative can be quite different. I mean, to give like a separate example, I know there was an issue years couple years ago about PLoS, or the Public Library of Science, which is kind of an alternative to Elsevier, in traditional publishing, you know, open, whatever. But, you know, there's their CEO had kind of a high salary. But I mean, CEOs have high salaries. I, you know, I it was it was kind of an interesting objection that people were making because I mean, what, what are you supposed to do not have a CEO. This is not academia. It's an institution outside of academia that has a different pay structure. And it doesn't have academia, the pay structure, it doesn't have academia incentives. And if people just were not internalizing that, from what I could tell, so can you talk about your experience at Elsevier, where academics were not realistic?
Well, I think the big place where you see that show up is where people don't have skin in the game, right? They people will complain About being forced to publish or forced to do that. And, like resent us because, you know, we publishers, like we, you know, sit there and we control their destiny? Well, I mean, that's sort of overlooking the whole fact that, like, it's other researchers that sit in all the study sections, and it's the other researchers that are the peer reviewers, and it's former researchers who were, you know, the editors, for the journals, and so like to talk about, I felt like a lot of a lot of the criticism was kind of like, somebody complaining to Twitter because somebody tweets, something that they don't like.
Yeah, you're just a platform
Yeah, it's, it really is, it really is a platform. And I think, you know, the big, like, reasons that you don't see, you know, innovative new technologies being developed and cures, getting out to patients, and like, the things that science is supposed to bring to the world. Those have to do with, you know, some of these misplaced incentives that are part of the system, which you would think like, okay, so like, this is a this is a thing that we need to fix it, but unless you can see the system from the perspective of all of the different parts, you don't know how, you know, pulling or pushing one is linked to other linked part
Yeah yeah, they're seeing the outputs, they're seeing the outputs. And so they don't understand everything that needs to go into the inputs. I mean, in some ways, it might be almost like, oh, well, I could fix academic science by doing this, this and this, and then it turns out that sometimes there's a rationale or reason to some of the things that they do, and it kind of like falls apart, right?
Yeah. So if there's one thing that like I could get people to take from this, is that, like, build it, and they will come is not the thing that's going to work in science, there have been so many platforms that people have built, to allow, you know, like, to get around, you know, the publishing, hegemony and blah, blah, or to change, like to get around the impact factor that change how, you know, research, you know, metrics, or, you know, how research has since incentivized and to get around publication bias and all of these things, and they've been built, and nobody uses them, because academia is super conservative, and building, yet another one is not magically going to fix things. So I have this sort of idea that what we, what academia needs.
Actually, actually, well, before you keep going, I actually want to set you up for the again, the listener who's not, you know, gone through academia. So I think what you're talking about here is okay, like, what are academics producing? What is their output. Now, there are some academics that do public private partnerships where they're producing something for the Mars Corporation, or, you know, some Engineering Corporation. But a lot of times what I could like if you're studying eyeless Drosophila using a generic example, your output your product is publications, and so this is how you get evaluated how many publications you have the quality and the impact of your publications. And so when you're saying people are conservative, this is a system that's been going on for almost a century, like a little less than a century, where this is how they're evaluated by their peers. And so people might complain about it. But this is the only system that they know. And, you know, when push comes to shove, they're not going to end up switching to some something else unless they're forced to. So when you say build it, and they will come, they're not going to they're gonna have to get forced, probably to switch out of this system. It's one of those things where everyone's is unsatisfied with it, but everyone is stuck in the current system and the first effectors are going to get penalized. This is even happening. This even happens with Mike Eisen's lab where he's the only publishes an open access. But people who've left his lab they end up publishing in the quote "glam journals", you know, because, you know, you can you can tell them for seven years or five years, this is what you got to do. But the incentives are really strong. And unless your Mike Eisen it's really hard to resist it. So I just want to set people up there.
Yeah, that's you're absolutely right. There's there really, you know, attractors in academia that have a whole lot to to explain for both the good side of what happens in academia and that it can, like resist political pressure and it can focus on the ideas as opposed to, like, having to think about the commercial outcomes are, you know, kind of being bought buffeted around by the you know, the, the social winds of the of the moment at but so that's the good side of the conservativeness. But the downside of the conservativeness is that we we know that like peer review is broken. Like there's, it's it's holding together for now. But so many people are being asked to peer review so much stuff that like, there's this greater sense that like, eventually it is not going to it's just not going to be sustainable over the long run. But we can't change it.
Well William talk about talk about why peer review is broke. I mean, a lot of people are not gonna index. So I can tell you so for example, my friend Thomas Chatterton Williams, when I told him a couple of years ago that academics don't get paid for what they're published, you know, that they pay journals, in many cases, uh, he just was his mind was blown, he had assumed that publications, were paying for these papers, talk about peer review, talk about like, how this publication process works a little for the listener.
Okay, so and this is going to maybe blow some people's minds, but you think, you think of an author as someone like, you write and you know, you publish this work, and you get, you know, you get it, people buy your work, and you get paid, it's not at all how it works in academia, in academia, like you get, you get money from a funding agency, because they think you are smart, are effective, or knowledgeable. And in exchange, like for you getting that, that grant funding in exchange for you having the freedom to just, you know, spend this money and be smart, you are expected to publish. So the publication is, it's an obligation that comes with like you're accepting grant funding. And it's not like - publication is, in science, it's more like, advertising the fact that you've done some work, and it is not the work, right? There's, because what you're what you're basically doing is, you're writing up a narrative summary of your work. It's not like a work of literature. That's why most scientists are totally fine signing over their copyrights to the publishing companies, because the... all they care about, once the paper is published, they're done. Right? They, you know, they're not out there promoting it, it's not going to there's, you know, the the reputation that they get from publishing this thing, and other people citing it, that's, you know, that's great and everything, but they care more about, like, where it was published, and the quality of the, you know, the brand of the journal, which has some true aspects and some kind of BS aspects to it, then anything that happens after that, so they're really just taking all of the research, they've done, the data, they've collected the experiments, they've done, the observations they've made, and, and wrapping a narrative around it, and then announcing, hey, this is what I've done with all this grant money that I've had this is, you know, and this is what I plan to do. So it's just, like, it's a totally different kind of publication from what most people think of, and there's been a lot of, of misplaced projects and design of systems, in academia, based on the idea that, like, published like, scientists, or authors, you know, I think, and, you know, this, this never really got off the ground. But I would love to see, like, a publishing service, where it's literally like a form driven kind of interface, you upload your data, you upload, you know, like your figures, you, you kind of, you know, you upload your citations, you say what you want to have, you know, said about it, and then that platform can take those atomic atomic pieces, you know, the narrative, the data, the code, the images, citations, whatever, and render them in whatever format the consumer wants, whether it's, you know, through an API that goes out to some platform, or whether it's, you know, on the phone, as, you know, as a PDF, or an Epub, or, or whatever. And there's some, there's some projects that are that are leaning more towards that. But yeah, it's so far away from the traditional, like, authorship metaphor, the further away you can get, the better I think, generally.
Yeah, it's a it's a very exotic land where you take certain things for granted, that are kind of the upside down of what you would expect. And then there's all these norms like your peer review, and then you know, the weird things happen in this world where A friend of mine just, uh, he's early 30s. He's early career, he just got his first request to peer review his own paper. Which is, this is the thing that happens, because science is so specialized, there's only like, you know, there are fields where only 10 to 20 people are available to peer review, and that it gets to the situation where everybody is in the same pedigree, like they come from the same, like, you know, grandparent researcher, and it's just a weird situation.
So you mentioned, you know, why is peer review broken and, like, it's, first of all, is struggling under its weight, because we're publishing more and more papers, and people are getting more and more specialized, and it's harder and harder to find people to put in the time to peer review. Because just like publishing, it's an, it's an expectation of an author, a peer reviewing other people's papers is also it's an academic norm, it's an expectation that you have as an author. And so it's, it's not, it was it worked really, really well, when you had, you know, back in the 50s, and 60s, when you had much smaller, more cohesive kinds of academic disciplines. But now with these massive publication platforms, trying to, you know, have a million papers peer reviewed a year, they just can't, you just can't find enough scientists to spend enough time to do a good quality peer review. And now that, that what that does, is it puts a lot of work on the editors. So others have to do a really good job of like, keeping the peer reviewers in check, making sure they actually, you know, do the review. And, you know, in the, it's raised a lot of questions about like, what do we do next? Do we actually move to a stage where people just put their, you know, their, their preprints write the publication that you write, but before it editor seen it before, reviewers have seen it, or, you know, before it's been like, made pretty by the the things that publishers do, just put that up, you know, on -in a repository like, you know, Archive Physics or Bio Archive in biology or wherever, and then have the review happen. After the things go up. It's a really attractive idea. But for one, you know, this old school conservative norm that like peer review is a thing that I'm supposed to do. When a journal asks it of me gets more things reviewed than would otherwise. So if you think about the total pool of people that could review papers, and how you spread them out across the total pool of papers that need reviewing, when you have a journal coordinated and do it, it spreads it out a lot more evenly than if you take papers you posted in Bio Archive and people self select, because you'd have some paper 100 reviews, and some that would get none,
right? Yeah, the Pareto Principle kicks in where it's like, yeah, 10 to 20% of the papers get 90% of the reviews..
Yep, yep so it's so good theory that it doesn't
Yeah, the journal, I see what you're saying. So what you're saying is like the journals are basically the journals are making people eat their vegetables.
Yes, yeah. They, and they, they keep reviewers in check, like I had to, in from the communications role. I worked really closely with our editorial team. And sometimes they would, there would be, you know, legal disputes among reviewers where like the reviewer, thought he knew where.... the author thought he knew who the reviewer was. And, you know, they like, want to try to sue them or they have, you know, try to pick a fight with him on social media or something. And, you know, it's - the editors do a really tough and very much thankless job people do not know how much how much like, editors are keeping - keeping the whole like scientific communication process together with like bubblegum and duct tape in a lot of places. It is, is really a tough, thankless job that they do, in that you might have two reviewers that are one of them puts in a totally incoherent, you know, nonsense review, you might have somebody who puts in a nice review, but you know it but or it's thought, a good review with a lot of good content, but does it in a nasty way. And editors kind of have to like figure out how to take all of this input into the best job in helping to improve the paper in the end. So, yeah, there's, but it's not going to last that's that's the thing that's fundamentally we're going to run up against is there's just not going to be enough people to do that maybe we move to two levels. So so like, you can see someone doing a statistical review where somebody else just looks at, you know, do these, you know, western blots look, right. Whereas somebody else goes through and looks at the experimental design, and it's bucketed up that way. As opposed to were it is now, like you have, you get the PDF, and, you know, you get a link to enter your review of the PDF, some, some publication systems are asking for things in a little more of a structured format. But I think that is, that's going to have to, it's going to have to change kind of had a hope for some of these, like online annotation systems like hypothesis, to really boost how people can do review, because you're not like reading this thing. And then kind of talking about your thoughts, because you could pick out certain specific things saying, you know, this particular like, laying in this western blot is, you know, looks funny to me. And you can review a paper in more detail more quickly. That way. So you could bucket it out that way, or what you can move to, you know, is a system that... And I'm laughing as I say this, because I know how much of a mess that is from the back end from Matt. But something that looks more like moderation on social media sites then peer review, where you've got like, you'd have this big platform, like, let's say, Bio Archive, goes and talks to, you know, Softbank, and says, Hey, you know, give us $100 million to own biomedical publishing. And Softbank says, yeah, here's 100 million come back to us next year when you've spent it all and they basically like, say, we will pay, you know, we will outbid everyone for all of the reviewers in, in in biomedicine. And this bring all of them onto that platform to review all the things. Okay. Yeah, that might be the sort of thing that that happens, where in which case, I think it's going to start to look more like moderation.
No no no wait, are you talking about a bio Reddit? I mean, it's like, like, basically, you're saying, like scientific publication could be replaced by Reddit. I mean, I'm being glib, but you know what I'm saying?
Well, so here's a, here's kind of a metaphor that I've been, I've been tossing around. That's particularly salient to me. Because of the work I've been doing at Quora lately is like, if you think about Quora, and you think about Reddit, Reddit is a forum where everyone's kind of own like, you know, equal standing, and Quora is more like, it has more like a have you think about, like, a course space is more like a platform, stage and, and, you know, audience kind of a metaphor, which is more like, like, imagine you're at a scientific conference, somebody's up on stage, they're giving a presentation, you know, but there's people in the audience that can ask questions, and, you know, discuss amongst themselves. I feel like that metaphor probably works better, just from like, the affordances that you have in the interface. Then the sort of like egalitarian just, let's all get together and discuss things sort of concept, but something like that.
Yeah, totally. okay. So I got to ask about Quora. You know, I think most of the listeners know what Quora is. You know, I'm just gonna like, this isn't a video so they don't know. You are a white male. I thought everyone associated with Quora was now Indian.
Well, so this is the really interesting thing about Quorathat I've learned is that you know, Quora has has Adam D'Angelo, former, you know, CTO of - from Facebook It has lots of you know, some of the best machine learning talent in the world for a while, like and and as far as I know, the still the case. In there are engineers who will turn down jobs at Google or Facebook to go work on the machine learning team at Quora, because they're just such interesting problems and such good talent that's there. Because they have all of this the site is extremely like, tailored and customed to whoever it is that show up.
Well. Yeah, actually, I gotta be. I assume everyone knows what Quora is, but just like tell the people what everyone is Quora is so that,
you know, right. So core is it is a platform for people to share their knowledge. So there's you can go to Quora, and you can you can ask a question. And you can like, Get specific people who use Quora to respond to your question by sending them a request. And you actually get a little blurb about what their expertise level is. And the system can recommend you, the people that are both most likely and most able to respond to your question. And this is where the parallel with peer review is really, really interesting. Because journals, the reason peer review is breaking down, like and not able able to handle the load is because they reach out to the most qualified people, but they have no way of knowing what the capacity of those people are. And so it's fascinating to see that like, oh, Quora, you actually had data on like, not just can this person, are they qualified to answer the question, but like, are they likely to answer the question, if, you know, if it's asked to them at this particular time of day, you know, or whatever, and so. So you can get your answers back that way, we also have a product called spaces, which, you know, again, kind of a, you know, think about going into a conference room, there's somebody on stage presenting, there's people that follow the groups, but there's, it's more about the owners and contributors to the space curating questions, writing posts about a specific topic, or, or, you know, thing that they have a focused interest in. And so that's the way that kind of like, Quora presents all these different faces to whoever, whoever shows up, there's a huge, like, kpop fandom on Quora, which I never see. But it is, like, if, in my use of the site, but if I go and look at kind of, you know, the backend traffic and stuff, it's huge, you know, there's obviously, like, a huge Indian user base on Quora. But I mostly don't see that stuff in my feed. Because, you know, I'm following science spaces, and some, like, you know, wine making spaces and things like that. And it's just not something that that pops up a whole lot. They use it very differently. You know, there's, they use it kind of more like a, you know, like a Facebook or something. And it's kind of more social and there
Yeah, for Indian people Quora has become a social media thing, right? I mean, this was my understanding. Yeah. And so they have like, just like, whole zones of, of Quoraare basically colonized by them. But you're saying that there's a lot of separation with all these communities.
Right, you go on there, and you like, you tell Quora what you're interested in, follow some spaces, follow some people in it, and it learns from your activity, what you upvote, what you downvote, you know, what you mute, et cetera. What, what to start showing you and you get this, your experience, of course, is totally different from like everyone else's, which really makes it kind of hard from like, a, a communications perspective to talk about Quora because what I see is so different from everyone else see. And just like with, you know, all the complaints that we get from Twitter about people saying, oh, you know, Elsevier is this, you know, research is broken for this or that reason, just speaking from their perspective, you can always complain some Quora people say, No, this is broken, because it's broken for them in a specific way that they've, they've started to, to use it. So it's really, it's really interesting. But
Yeah so um, you know, I, I go to Quora every now and then. Well, I mean, they just comes up in searches, and I read it. And so I've actually never gone to Quora. I mean, I have an account, I'm logged signed in, but I've never gone to Quora. i Not that I know. Except for like, when I'm searching for something like sometimes, like, there's a lot of things where actually, my name comes up, where people like will cite my blog post or something. Like I just like, curious what people are saying, because, you know, here's what people are saying about me. Right? But so like, I've just, like, started to explore this, like, why while we're talking and so yeah, like, people should definitely check this out. This looks interesting. And, you know, my friend David Middlemen who you who you probably know of, like, he's always been fascinated by Quora and what they're doing with information. And so I guess, like, you know, this is kind of like a segue to the next topic that I want to talk about meta science, communication. You know, like, obviously, you have your professional interest, but I have always felt like you, you know, like, you follow me, you follow us like star Codex, like, you know, Scott Alexander, this sort of stuff, that you think deeply about how information is coming out of there. And so you said something about the reproducibility crisis, that you were having a discussion with someone many years ago and it was a little disturbing, like your, your ultimate conclusion, but um, can you can you talk about that a little?
Yeah, so obviously, I can't leave the topic of Quora without mentioning that we just launched monetization features. So you can now get get paid for, for answering questions and having spaces, as opposed to doing it, you know, out of your - the goodness of your heart. But yeah, the the reproducibility initiative, I worked with a - I worked for the Center for Open Science, which is a meta science, an organization that kind of got started...
Is that Nosek's thing?
Yeah, Brian Nosek, Brian Nosek's thing. And, you know, he, he kind of got a lot of his, his start around the, the crisis that psychology was going through, where there's just so many of these fundamental things that they had been publishing and building on, turned out to just be phantoms. And not really, based on a solid evidence base, some of it was almost like, you know, like, [inaudable] , and some of that stuff was deliberately fraudulent. But they were interested in funding, other meta science kinds of initiatives outside of the social sciences. So myself and the CEO of the startup called Science Exchange, which is an outsourcing platform for getting scientific experiments done. So you can have a virtual lab and that kind of thing. And I think they've pivoted a little bit away from that to working more with like pharma now, but at the time, they was kind of like, the Airbnb for scientific instrumentation, you know, you have, like your flow cytometer, in your, your research group, probably isn't at 100% utilization all the time. So like, maybe that particular instrument can take in, outside work, you know, maybe the person who runs it can, can take outside work, and get paid to run it in the, you know, the idle time, that was kind of an idea anyways, like, so you have a ability to do research. And then you have, like, from my side interested in about research, metrics, and reproducibility, like, here are all these things that are really, really highly cited in biology. I, I wonder if they're true. And there have been some, you know, there were some papers published, looking at you know, breast cancer biology, looking at ALS, looking at, you know, cardiac interventions, saying, like, look, you know, all we're just really struggling to build on this preclinical research to come up with a drug or therapy that like, actually makes a difference for patients, because we have all of these papers with hundreds and hundreds of citations. elucidating the mechanisms by which this disease occurs, but none of the, like, drugs actually work. And so like, what is it that we were not getting here? And so I thought it'd be interesting to go and take some of those research papers and run them and like, redo the experiments, break the papers into pieces send off, like, can Okay, you know, you know, western Colorado, you know, flow cytometry facility, hey, will you go isolate this population of cells and stain it with this, these markers, and, you know, just give me the data back. And then you work with another organization to do you know, some histology and just kind of go through these papers, re gather the data. But from a widely spread group of labs, none of whom really know what each other's doing, which means none of them have that publication bias where they're like, we're going to suppress, you know, some findings, or we're going to, you know, like, edit the data a little bit, or we're going to keep running the experiment, to get the finding that we want, none of the left hand doesn't know what the right hand is doing. And is that going to come out with a different set of results? Now, obviously, to do all of this work, takes a lot of money. And I was running around trying to fundraise and I spoke to, you know, like the assistant director at the NIH and said, hey, you know, what, like, the NIH should be funding a lot of this stuff because you're dumping in unknown amounts of, you know, billions of dollars - and for listeners who don't know, the NIH funds 99% of biomedical research in the US. It's not the pharma companies. It's not you know, the industry of any sort that's just a tiny minority
But I was talking to him and I said, Look I mean, how much further with these billions of dollars that you're allocating, to, to basic research go, if you shave some of it off to do some, like meta science to understand how to make the research that you're doing more - that you do do more effective. And he said, You know what, that's, that's a really good idea. I wish we could do that. But we aren't going to, like, make a whole lot of, of, you know, noise about the fact that our research is not really reproducible, because that's really going to feed in all of the, the denialists and cranks, and the peep, like the anti Vax people you have today, or the climate denialist or whatever. And so it was just like, it was a great idea. He thought it would like really move the science forward across all the fields that they publish, but it was just too much of a political hot potato for him to touch. And I thought he was I thought he was wrong. I was like, No, this has got to, you know, like, how could you be so short sighted, this makes research better. We knew that getting some funding, we work with a Center for Open Science. And one of their funders is to Laura and John Arnold Foundation. They contributed some extra money, we were able to get this stuff funded, but nowadays, you know, hearing, just like what COVID has really made extremely salient to me, and I think a lot of people is that, you know, Larry Kaback was right. The people don't trust the NIH, they don't trust the CDC or the FDA. And there's this distrust, you know, feeding distrust among people who think they know enough to to know better, but actually don't actually does more harm in the long run, and it's - that's a real, real bitter pill to swallow. And that Twitter thread, you know, there's lots of people who are really just kind of making making my point. And that's not to say that people don't have valid reasons to distrust some of the things that CDC has said they got some things wrong about the aerosols and such. But the overall point is that trusting these people to do a good job on the things that they're, that they're doing outsourcing, you're, you know, you're, you're thinking to them, for most people who don't have the ability to dig into the primary literature and figure things out for themselves, or even the people who do but who just don't have the time to do it. It's a better use of your time, it's, you're better off overall, trusting them, even though they're wrong occasionally, then it would be to just say, Oh, they were wrong about this one thing, therefore, they can't be trusted. I'm gonna throw out everything they say.
Okay, well, I mean, the high level I agree with you. But I do think that there's some differences with, say COVID-19. And I don't know what it's like too much of an anomaly, say, like, the selection neutral ism debate in evolutionary biology, like that's been around for 50 years, there's a lot of data, there's been a lot of, let's say, epistemological, like a competition between theories and models and hypotheses. I think the problem like let's talk about COVID-19, because, you know, this podcast, like, I tried to make it evergreen, so I guess this is gonna be, like, not as timely if someone listens to it five years from now, but you know, what, that's fine. I think the issue with COVID-19 is, this is a new thing. And a lot of people had priors and base rates. So you know, I was I was worried in early February, and I can tell you, one, all of science, Twitter was ignoring it. And two, you know, you be called a sino-phobe, but, you know, why do you care about this? Is it because there's, it's from China? Like, are you a xenophobe? But, and so, like, that was like, like, you know, even until, even until like, actually, like, on February 24. Um, you know, I got a DM from a friend and I'd been getting direct messages on Twitter, from a friend about like, what do you think and I'm like, I think you should, you should hunker down, you should be scared, like, stuffs gonna come down the pipeline. And finally, I just decided, you know, I don't care if I seem crazy. I tweeted out like, you know, I tweeted out my response, and I had multiple scientists tell me like, Okay, I was thinking the same thing, but no one's saying it. So I thought maybe I was crazy. So like, you have this like, whole thing where I think, okay, the authorities seem like, you know, like, the whole mask thing. The mask flip. The aerosolized thing. Also, like this is like scientists get mad about this, because they're ideologically pretty uniform. No conservatives going to listen to public health after the whole BLM thing. Because they were not they were not saying any of the same things. When they were the people protesting earlier. In fact, they were like criticizing them making fun of them, saying that they're a public health threat. And then they say, Well, this is more important than that, like, Okay, I got it like, I mean, if you're conservative you got it. You know, you know, like when push comes to shove their politics more important their science. So this is I think I just want to say like there's special things with COVID-19 that made it kind of, and in other countries it's not as culturally polarized by the way, a lot of these issues are not as culturally polarized. They do have protest movements in places like Germany and Australia about the lockdowns. But it's not as ideologically polarized. You know, in Israel, it was a conservative government under Netanyahu that was doing the lock downs and a lot of the labor people, but also the ultra orthodox Jews are angry. So it's mixed United States is a very special country, in a lot of this polarization. And so I just wonder if, like, with COVID, we're kind of seeing a situation where, you know, trusting the authorities, I mean, if you trusted the authorities on mask, which a lot of people did nobody bought masks, you know, except for people in the know, like, I have friends who got N95s and other stuff early, because they were just this is BS, you know? So I don't know, I just got to put that out there. Because I think like, you know, when people say like, they don't believe in evolution, well, that's different than saying like, they don't believe the droplet theory. You know, whatever. I mean, like hygiene, hygiene theater still going on, people are still probably saying that they're doing deep cleaning, even though it's totally useless. And we've known this for a year, over a year, probably.
Yeah, no, you're. So you're absolutely right, that the CDC blew the thing on masks. And, you know, like we, you know, I wasn't there, I don't know that the conversations that happened, they said the wrong thing. One thing that I do know from like a spokespersons perspective is that like, the number of people that hear what you say, directly, is maybe like, you know, 1/100th or 1/1000th of the people that will hear your message. So what you do when you're putting out a message is that you try to think of all of the ways, all of the different people that are going to hear it all the different ways it's going to mutate as it moves along. And you try to come up with a message where the core thing that you're trying to say will make it through all of those, you know, all of those mutations. And you - obviously, like there's so much you just can't know, and some people are better at it than others. They could have been a lot more clear about the mask thing. I - they could have been a little bit more early. I think the the BLM stuff doesn't show the the hypocrisy, really of the CDC, I think so much is all of the the media and the coverage around it.
Well, I mean, there were 1000 public health professionals that side. You know, I mean, you know, it's not just me, I think the media and the academia. Like I mean, me concretely, when Trevor Bedford wrote about the possible risks from the BLM. He was pretty frank that he was kind of scared to do this.
Yeah. And, and he should have been, right, because people are not in their right minds. And I think when you combine the fact that like, someone's not in their right mind, they're extremely emotionally motivated. They don't have all the information. Well, if you're in that state, shouldn't you be kind of thinking twice before thinking that you know, that you're going to second guess, you know, an authority figure? Like, obviously, they're wrong about some things. But if you sum across all the things that they have said, they're probably right, more than they were wrong, and they probably would be more right than you were, were you the ones making all the same decisions they did. Right. That I thinkyou know, that's what it comes down to.
Yeah. And like, you know, in the defense of the authorities, you know, I mean, most doctors were totally wrong, because they were subject to base rate fallacy, but you should still go see your doctor. You know
Yeah, it's like picking stocks, right, you know, everyone thinks they can beat the market. And occasionally, you can, like, I, you know, I, I bet on Moderna when it was cheap, and did you know, beat the market there, but, like, if you sum up all of my decisions, you know, all of my decision investing decisions over my life, it's probably going to be about the same as if I, you know, chucked it all in, you know, a market tracking mutual funds, just because that's, you know, I don't have any special knowledge. I had, I had a little bit of an edge when it came to Moderna. Because I was like, Oh, I could see how this would be a platform. And I think that their early clinical data is actually looking pretty good, you know? So a little bit of an edge there. Where you do have an edge right then yeah, second guess all you want, right? Like, I think most people don't have an edge in in all areas, so that's kind of where you have to have some like, epistemic humility. And again, when people are really, really emotionally riled up, they're not in the right minds, they're doing things like, you know, going out in the street and chanting in mass crowds in the middle of a pandemic, right? That's not a thing that you do when you're like, you know, calm and rational, and, you know, considering all of the... that it's a decision made from emotion, I'm not judging it one way or the other. Yeah, but it's an emotional decision. And if you're doing that, you're not in your careful, let's consider all the best, you know, evidence and make the, you know, the best decision we can about other aspects of your life too. You know, that's where they say, Don't you know, invest when you're, you know, when you're hungry or when you're upset, right? You're not going to be making the best decisions. And so I think, knowing when you when it's okay to outsource your decision to other people is a skill that skeptics particularly need to practice, because their priors are like, you know, I'm being lied to, and these people are misinterpreting it are, they're telling me something that they're motivated to tell me, you know, like, and that's where you get all of this denialism and skepticism about like, Oh, it's just, you know, the, the drug lobby is, is telling us that, you know, like, Chloroquine is not going to work, because it's cheap, and there's not a patent on it or something, right? And so you really, really have to be careful. If you want to be a skeptic, you have to be particularly careful to know when you should trust the authorities. And that's really, really hard for people to do, who are just kind of in this, like, default skepticism state.
Mmhmm mhmm. So, um, you know, I guess the last question, uh, well, I mean, I think I know what you're gonna say, but I mean, are you optimistic in terms of building a better world through knowledge and information? Like, do you? Do you see things changing in our lifetimes? Like, I mean, does it get better? We're having a lot of stuff about disinformation. And, you know, all these memes and the denialism as you as you say, which is, you know, a term that comes goes back about 20 years now. And, you know, I don't want to go into all the conspiracy theories that are out there, but I mean, there's a whole, you know, culture around stuff like Q anon and anti Vax, and all these other things. So I mean, are you still optimistic?
So I, I really struggle I think, overall, I'm optimistic that humanity will pull through ], you know, our trajectory look, looked over, you know, a long enough timeframe has generally been up and to the right, in terms of all kinds of metrics, this is where like, "our world in data" is great, kind of like, you know, palette refresher to look at, because things are getting better, you know, childhood mortality is going down. And like, you know, the rate at which, you know, we're dying of, you know, car crashes are down, the violent crimes going down, all these were generally, overall, doing better. The thing about this is that this isn't a thing that just happens. And it isn't a thing that everyone is uniquely capable of making happen. There are specific people who, you know, feel a calling to make a difference, and a certain area, and they answer that call, and they really devote themselves to doing it. And those are the ones that make the difference. So it's, it's easy to think that with like these - in the day, where we have these, like, massive Twitter hordes that will descend upon, you know, the evil doers and cancel them for whatever, you know, mischief that like, you ...
Oh I know about those...
Yeah, so do I trust me, but I imagine, you know, a lot of listeners here can can identify, but the thing is, there, it's these it's easy to kind of like, let the bystander effect come in. And you think, Oh, well, like, I don't need to do anything. Because, like, there's just going to be, you know, like, somebody will take care of it. are, you know, the government, you know, there's agencies whose job it is to do these things. And if you trust the authorities, maybe you think like, yeah, I shouldn't so that's kind of the flip side of what I wanted to say is, yeah, trust the authorities, but if you feel called to do something, and You have enough epistemic humility to know what you know and know what you don't know. And this is a thing that you feel like you are uniquely qualified in some way to contribute to, then you have to jump on it. Because it's these, it's the people who have found that calling and took the opportunity to have made the difference every single time. And if we can get more of that, then that's what I'm really optimistic about.
All right, um, you know, it was great talking to you Will - you know, I'll see you online and IRL, hopefully soon. You know, as we're recording, you know, we're having another wave. And, you know, I think, hopefully, by the time this is posted, we'll be done with that, trying to look on the bright side of things. You know, we live in a world of information. You know, your company Quora's trying to navigate through it, I think that's great. We live in a world of information has transformed the 20 years we've known each other a bit online and, and, you know, it's some of it's for the good and some of it's for the bad, but we just need to figure out how to optimize things and I do appreciate your Pinker-ite viewpoint that it's getting better that we can have enlightenment and you know, you know, we're putting the past behind us so hopefully I will see you soon and thank you for your time.
