How AI Is Being 'Openwashed' and How to Fight Back
9:25PM Sep 21, 2023
Speakers:
Devin Coldewey
Darrell Etherington
Keywords:
ai
labor
signal
open source
infrastructure
talking
open
data
read
systems
model
requires
heartbleed
running
set
facial recognition
company
form
human
work
Maybe founded a company or two. We've got some great Pyramyd coming up here on the main stage before the reveal of the TechCrunch. Startup battlefield winner later on, but how AI is being open washed. That's an interesting term. I heard a little bit about this backstage. As our next guests and MODERATOR We're discussing what's coming up. But basically, AI you know, a lot of people are claiming it's open, including one very famous company with that right in the name. We'll hear about what that word means to some people, and whether it has meaning at all anymore in the context of AI from Meredith Whittaker, who is president of signal, and also co founder of the AI NOW Institute at NYU, and she'll be joined by your moderator, Devin. Coldewey. Welcome up.
Well, thank you for joining us again. I caught your panel with Zack this morning. Cool. Yeah,
that was nice. That was excellent. Exactly. The
back is the best. Yeah, so. So we're actually we're here to talk about open AI for once, not the company ban. But the concept of open AI. So you and your colleagues, David Gray, Witter in surmise West, yeah, this this, this article, it's in preprint right now, long article, but I did read it. And it's it's about how the AI industry the tech industry is sort of attempting to open wash the the sort of natural and emerging monopoly over the the resources that make AI possible. So before we get into I want to ask, What prompted you to tackle this topic in depth like do you not have enough to do is signal
Yeah, well, all of my scholarship is driven by annoyance, okay, that makes so there's something wrong on the internet. And I have to then write about it. I then have to sense it. You know, I can feel it in the room. Someone's going to be wrong soon. And then I'm going to have to spend my weekends you know, writing along print preprint. I also I enjoy understanding things. So one of the things that I started to notice along with Sarah and then we connected with David who was doing some work on open source in his PhD, maybe about a year ago, is it suddenly there was this sort of crop of dc dc Think Tank pundants who were publishing articles about or you know, reports, and making statements about the benefits of open source AI the benefits of openness and AI. And you know, AI itself is more of a marketing term than a technical term of art. It describes a number of heterogeneous systems often that we can't inspect because, of course, they're proprietary. So we don't actually know what they're running. You know, it's a, you know, spreadsheet formula behind the, under the hood or what have you. And, you know, I wasn't clear how does open source as a fairly, you know, Arcane licensing regime, and set of ideologies and principles actually map on to the material reality of AI. So I'm sort of scratching the surface looking for, you know, explanatory footnote, what are the definitions we're working with? And over that time, we saw this sort of crescendo of hype around open AI the claims made about openness and AI kind of inflate in importance in the discourse around AI. So it was really like, Okay, let's try to map what we're talking about materially, what can be open about AI? What is the difference between the open source ideology and the sort of principles that were sort of formed around software development in the 1990s when these concepts were birth and the tech industry and you know, the AI as a modular system that requires huge amounts of resources, both to create and run, you know, how are these different and the same and you know, the TLDR for those of you who don't want to read a 30 page, academic paper,
it's very interesting. It's,
I read it actually in prep for this and I was like, Oh, this is a banger. It was pretty good. I'm just, um, but you know, AI. You know, AI requires huge amounts of resources. The term open is being applied to a wide variety of kinds of AI offerings and you know, I reinstall them on at hugging face documented this very well another paper that she wrote, but you know, anything from offering an API that lets you try the model is being called open source, you know, to llama, which is kind of a you know, Facebook's llama to they released it with a you know, bespoke license that has nothing to do with the standard open source licenses that are sort of endorsed by the Open Source Initiative. Not very permissive, you know, to things like infrastructures that are being you know, systems that are being created by nonprofits like a Luthor AI, which go, you know, as far as possible to make the parameters open to make the datasets open to document the provenance of their datasets and to make the train model the, you know, algorithm that has been trained on whatever the data is. And have gone through their calibration and evaluation process. You know, the code for that model actually open and reusable under sort of standardized permissive licenses like Apache two. So there's all of these are clustered under the umbrella term open source AI and are being leveraged in a very fierce regulatory debate you know, in part to try to exempt certain forms of AI from regulation and in part in ways that we described in the paper but are a little too nuanced to go into on on stage here, you know, as a form of kind of corporate capture, which is not, you know, which has actually happened around open source projects in the past. You can look at Android and Google, right, they were able to leverage huge amounts of free labor, but they didn't actually the sort of break up the monopoly of mobile operating systems, et cetera, et cetera. You're seeing a similar dynamic here. So our conclusion was that you know, there are you know, in the case of say Eleuthero AI, and things like GPT Neo, which is a model they released, there are, you know, maximally open versions of AI and what does that mean? That means that they go as far as possible to be transparent. So the code, the parameters, the provenance information, etcetera, are all made available for scrutiny, that can help with certain forms of evaluation, you know, kind of auditing but you know, is not, not everything we need to evaluate the implications and the fallibilities of these systems. They are also reusable, so you know, Apache, two very permissive license so you can take it off the shelf, you can, you know, mod you do what you want, you can reuse it, pretty standard open source stuff, and you can extend it so you can build on top of it. You take these big pre trained models, and then you tune it, which is a word that means you kind of do a little bit more training that then sort of targets it for a specific domain use, or what have you. But what does that mean? Well, that doesn't mean that the playing field for AI development is leveled, right, because the massive amounts of infrastructure that are required to actually create these models from scratch or of course not open, right, the data may be open. So you know, GPT Neo, is trained on the pile, which is an open data set that was also compiled by Luther AI. This is you know, one of the datasets that are being used a lot in AI research, but you can't just take the pile off the shelf and like use it to train it right? You don't just put it into, you know, some data framework and, and it's done. There's a huge amount of labor required to sort of curate this data, often sort of adding in other datasets and making data that can actually sort of inform these models, and create models that will behave in a way that is, you know, marketable are useful, right? So that labor is not open and the data is not ready for use in any meaningful sense. Then, of course, there is the labor needed for what I'll call the calibration process, which is often referred to as reinforcement learning with human feedback, which is a made up jargony term that sounds technical but actually refers to a kind of colonial practice of paying precarious labor, usually in the majority world, to make these systems to train these systems by human feedback, to behave in ways that I would say, fit kind of business safe. Liberal discourse. So we do not want them regurgitating the vitriol, the racism, the misogyny, the nonsense that is reflected in the data that they are trained on, right, like a lot of the, you know, they're trained on Reddit. They're trained on 4chan. They're trained on the web, right and so you need to actually go to painstaking lengths to sort of curtail those tendencies in these models. And that requires an irreducible amount of human labor working under very traumatic circumstances. And we can talk about, you know, a number of the Kenyan workers who've actually formed a union as an attempt to try to push back at some of the labor conditions are required to get these models into a shape where they're usable. So that is not open. And then we get to I know I gotta get started, I know and then they're going to accept one line. And I'm going to be like, but that's not. And then, of course, deploying these models, so you can then release the code for these models, you can release everything, but everyone in here probably knows more than most people in the world that like inference is also really expensive. So actually deploying a model even if you take it off the shelf, you know, from some, you know, open, open, you know, hugging face library, you're going to be having to spend a huge amount of money to deploy this model in production if you want. And there's an estimate that I saw again, we don't have these numbers, because they're kind of like hidden in cloud contracts. They're not available to academics for public scrutiny, which is another problem but if you actually want to, you know, there's an estimate that running chat GBT for a week, that Microsoft ad for their Azure API costs more than a training run of GPT. So we're talking about huge amounts of money to actually deploy these models, which of course are not open either. And sort of all of this means is diverge is really radically from the assumptions and ideologies that we immediately think of when we hear open source because of course, those were informed by our understanding of software and our understanding of, you know, what openness does with software which enables many more things than enables with these sort of, you know, AI, which at its heart requires this sort of concentrated power and infrastructure. So that's the paper.
Yes. No further questions. Thanks for coming. Thank you. So let's, let's let's let's dig into a couple of these will not but seriously, you should read the paper because it goes into a lot more detail on these and there's a ton of references that you should read. But I want to talk about a couple of things that you didn't mention or only mentioned in passing. For instance, you have meta and Google who have these, the sort of the most popular frameworks in pytorch. And TensorFlow, this isn't the first time that dominant companies have essentially sort of seeded open source with software that is favorable to them down the line. And then they've said this on the record, they're like, oh, it's very good for us to you know, be present in this open source position, which is not in itself a nefarious thing to say, but you're like, but this isn't the first time this has happened. And it hasn't turned out well before. Right.
Well, I mean, I think in meta and Google are slightly different because Google is obviously an infrastructure and cloud company and Metis trying to figure out its role without an infrastructure business model. But you have you know, meta developed and released pytorch. Google has TensorFlow. These are, you know, this is how AI gets built, right? Anyone taking a course at NYU, you know, you're just using those tools. That's how it works, you know, and your kind of scholarly definition of infrastructures. It's something that blends into the background and you take as a given, you don't actually think about it. Right. So this is really, you know, kind of shaping in creating the water in which AI research and development swims. And why would meta do this? Why would they, you know, spend a huge amount of money on this. Why would they pay the core maintainers employ them? Well, because they get to define the terms of how AI looks and they get to make you know, any of those academic models and most academic AI is sort of built using pytorch right now can then snap into place in their production systems very easily, right. So they maintain a sort of, you know, natural interoperability between everyone developing AI which uses their, you know, their frameworks and their libraries and their production system. So they're able to you know, implicitly control the direction of AI research, you know, put more development effort into certain parts of those frameworks, other you know, others into other parts and sort of, you know, by controlling the infrastructure by controlling the temperature of the water AI research swims in, they can control the you know, the conditions more broadly. And they can benefit from that by just taking what they can kind of harvesting what they want from that research, and putting it into play in there. And I read through hours and hours of earning earnings reports and they're very, you know, we quote them here, but they're very clear that this is your core to their corporate strategy. around AI and part of why they're sort of, I think, really, among the big players, they're the ones who are doing the most dissembling and the most sort of sleight of hand with the term open, right? They've realized this benefits them to sort of champion openness. Openness has a sort of halo of virtue around it as a concept. And so they're running hard with openness, even though no one who understands these things actually take seriously that llama two is an open model. And there are bigger questions on whether AI can be open at all in any meaningful, you know, as compared to what we think of when we think of open source software.
And so this is, I think, that it's important to say why it matters whether it's fake open or real open. And the the example you give later in the paper is with Heartbleed. That was the sort of oshit moment insecurity where they were like, wait a second, why, like why does half the world's servers use this protocol and it's, it's been funded by like, $5,000 a year and a bunch of volunteers and the entire world runs on it. And so when you have this critical vulnerability, it affects everybody and people are like, Oh, should we actually act to support these things? So what does the ocean Heartbleed moment look like? For AI? You think? What's that? What is that going to look like?
I don't know that there's a clear analog because Heartbleed like, if we turn over the rocks and the infrastructure that, you know, network computation runs on, it is gnarly, right, like NTP is maintained by a guy on a boat who keeps his coat and BitLocker. Right, like, that's not true anymore. But they used to be true, right? And I was actually working on a lot of these like core infrastructure initiatives for a long time. And like if they can take it for free, they'll take it for free, right, like open source doesn't mean people get paid. It doesn't mean people want to read your code or review their code. It doesn't mean anyone's gonna go in there and spend any time on it. It means that, you know, it's open to scrutiny. And there's a license that is permissive that lets people sort of grab it and use it and that you know, at the lower layers of the stack, the reason we have it, you know, networked in network computation is because things have to be able to interoperate, right, they need to be able to connect to each other. And if you're not both using the same spec, things don't connect. So like the lower layers are all of these open source, sort of core infrastructures we built on top of them, you can't, you know, change the bottom while the top is still running. So it's, it is gnarly down there. But I don't know that there's an analogue with AI because again, I don't think what we're talking about is open source AI. We're talking about a few you know, certain transparency attributes, certain reuse. Attributes, and certain sort of extensibility attributes. But we're not talking about, you know, that doesn't necessarily make them core infrastructure in the same way that these sort of, you know, SSL, which is the system that experienced the Heartbleed vulnerability, and others our core infrastructure. So I think it's again, I think it's a misuse of the sort of ideological halo that attends open source. And I think, you know, there's a real question over whether we should even be calling open AI, open source or if we should be thinking about something like open parameters or using more precise language that signals what we're actually talking about.
And that seems like something Irene has been talking about in her research. Yeah. Yeah. So I think Is it a coincidence that the companies that are investing the most in this are also sort of at the forefront of the surveillance economy?
Well, no. AI is a surveillance technology and it requires the surveillance business model. It's an exacerbation of what we've seen since the late 90s. And the development of surveillance advertising. So it's not you know, this kind of the Venn diagram is a circle, as I've said, you know, both AI, what we're calling AI now, which is this sort of large scale, more data more compute, requires more data where I've already seen the sort of the kind of rush to mass surveillance and the kind of, you know, contested relationship between, say, Reddit and, you know, the Internet Archive and others and the kind of desire for data. So AI is a way to, you know, I think I think it's a way to sort of entrench and expand the surveillance business model. And then the use of AI is also surveillance, right? You know, you walk past the facial recognition camera that's instrumented with pseudoscientific, emotion recognition, and it produces data about you right or wrong that says like, you are happy you are sad, you have a bad character. You're a liar. You're whatever. Right? And these are ultimately surveillance systems that are being marketed to you know, those who have power over us generally are employers, governments, border control, etc. To you know, make determinations and predictions that will save shape our access to resources and opportunities. I
want to touch on the libre thing that you mentioned earlier in the sort of the colonial dynamics you mentioned. There's a you have a great quote from from Paul Venrock. I'm sorry if I get his name wrong, who's the founder of Ghana, NLP and open source project? He says if African AI and ML researchers are not careful, this new open source movement championed by the richest global tech companies will become a mechanism for continued exploitation of our human capital and continent. So how do we like Paul said it like Yeah, I mean, like So what So what can we do? I mean, from from this side of the equation, and I know that there like people are organizing over there. Obviously Paul is doing something good is there anything we can do on this side to sort of improve the transparency or make sure that you're they're not using sort of this shitty exploitative labor?
I mean, transparency without power is a flex, right? If I can't do anything about it, you're just telling me it's happening. Like what are we going to do? Right, like so. You know, the exploitation of labor again, is constitutive of the, you know, the large scale AI construction process. There is no way to create these systems without people labeling and curating the data data. And when we're talking about the data, we're talking, you know, like this is the trillion parameter data sets right? You know, trillion tokens, token data sets. This is a lot of data. It requires a huge number of people, whether these be Mechanical Turk workers or you know, people working for companies like sama in Kenya, which outsource this work to low wage workers in Kenya who needs to sort of you know, effectively their labor creates the intelligence that AI is then marketed as embodying
the truckers were actually outsourcing their labor to AI. Yeah,
I mean, it's like, God bless them. I don't know. I might get it. There's no way to make these systems without human labor at at the level of informing the ground truth of the data and at the level of this sort of, you know, this calibration process that requires, you know, the reinforcement learning with human feedback, which again, is just, you know, kind of tech washing precarious human labor as a term. So it's, you know, it's 1000s and 1000s of workers paid very little but you know, on Moss, it's very expensive. And there's no other way to create these systems full stop. So I don't you know, like, how do you deal with that? Well, you you know, there's unions, you push back at, I think we expose the labor conditions that are required to make AI work, and recognize that in some ways, you know, what we're seeing is kind of a Wizard of Oz phenomenon. When we pull back the curtain, there's not that much that's intelligent, there's not that much that is, you know, automated that there are, you know, people who have to do this at every step. And then, you know, let's do some back of the envelope calculations like how if we actually valued labor, if we actually valued, you know, ensuring that these people were paid decently were able to have like, a lovely life during their time on this planet had the resources they needed, would AI be economically feasible, right. And that's putting aside the fact that a lot of the labor for the calibration process is highly traumatic, right, as people being exposed to some of the nastiest stuff on the internet and having to sort of, you know, watch it, see it again and again, as they are sort of the human buffer for that nastiness, who informs the system that this is what we do not want to see. And there's some pretty heartbreaking videos you can see of some of these workers who were like, I lost my friends because I just broke down. Right? I was seeing this thing over and over again, it was too hard and there's no support for that. And I think we need to, you know, that's something that you know, higher wages and, you know, setting a floor for employment etcetera, will not solve and is again constitutive of getting these systems to work in any way that would be acceptable in a broader setting. It seems
like there's a lot of echoes of the moderation problem that we saw over the last 10 years.
Dealing with all the trash and at some case, there has to be janitors. And you know, sadly, janitors don't make that much money aren't that well respected even though nothing would work without them?
Yeah, totally. And so, this kind of, we have a totally different sort of side of this happening right now in Hollywood with the sag AFTRA strike where a part of it
is my favorite piece of AI regulation is
to bring it up, so it will tell them how does this How does this analog comparison where how does this fit into the sort of like, the backlash against AI systems or how we should understand this as as an effective way to push back against the idea of functional or ethical AI?
Yeah, I mean, I don't like in a sense, we're not really talking about AI, right. We're talking about like, who gets to use AI? And who gets AI used on them? What are those consequences, right, because I like the WGA strike didn't start as an AI strike, the sag strike kind of followed on once they realized like, oh, the AI clause is the one they will not budge on. Okay. Something's going on here. Right. And I've talked to some of the folks who are doing that. They're very smart, and they were like, Okay, well, this is clearly the issue. We got to look more into Oh, shit. Okay. And I think that, you know, what we're looking at there is a justice issue, a labor issue, and that AI is being used, like automation has been used since the Industrial Revolution. And this is a very well documented history as a pretext to justify degrading the working conditions for creative labor. Right. And so what the writers that I've talked to very astutely predict is that if we let you know, I'm saying GPT start writing films and TV shows, which obviously, it can't do in any quality. We will then you know, lose our job as writers and there'll be some shitty gig platform we'll have to log into every morning to you know, you know, underbid our peers to go in and be a low paid contract worker, AI editor, right, they'll still be doing the labor of like, making sure this makes sense. They'll still be contributing their creative labor, but there's just now you know, a pretext that has been established to, you know, consider that labor low skilled and thus to justify paying people less for doing that labor. And I would, you know, I would suggest actually like reading some histories of the Industrial Revolution, Brian merchants book on the Luddites is very good on this, but like, there's a rich history of people sort of clocking the fact that automation itself was not the problem. It was the way automation was being used by those who control that to justify undermining workers is the actual problem. And that remains true now. And I think that that's what we're seeing with sagen WGA. And that's also why I think this is really the frontline of meaningful AI regulation, right? Because they're not waiting for a guy who really wants to, you know, meet an AI CEO and Congress. They're not waiting for Chuck Schumer to make up his mind. They're actually just taking it into their own hands and being like, No, you're not you're not coming at us with that. You're not going to leave my family without an apartment because you've licensed the Azure GBT AI and now we're all watching, you know, you're gonna produce even worse streaming content on Netflix and the 50,000 other platforms I have to subscribe to because the internet made TV so much better. Yeah,
it's so much better now. I have to say, your mere mention of the Luddites? You can tell me about the book, maybe but just back maybe backstage but the the Neo Luddism kind of thing is very cool, because I love the idea. They're misunderstood. It's like they didn't go smash the mills or whatever, because they hated mills. They were like, No, these are the tools of our oppressors and we are smashing the oppressors, you know, sort of, by proxy.
I mean, it's so wild. When you start reading history, there's nothing like where you're like, oh, wow, the received wisdom really is upside down a lot of times, like their critique was like, we should be able to share in the benefits of the mechanical technology which is enabling industrial production at scale. Right? Like there's no reason that this technology should only benefit like one guy who laid claim to the land and built a mill, where the workers that work this land forever. Why wouldn't we get a piece of that? That was it right? Like sorry, like please just clap if you think that makes sense. Some people think it makes sense. Yeah. Please just clap if you didn't clap.
Okay, so but
like what? I guess Yeah.
All right. So well, yeah, we'll have some more audience participation in a second. We only got a couple minutes left here. I want to ask. So this is gonna sound like a very stupid question, but it's actually very smart. Does the signal use AI and machine learning in any of its products or developments or will it No,
that is a good question because I hate it. So we have a small on device model that we, you know, didn't develop, we use it off the shelf. It doesn't. Don't worry. Just reading but you know, it's not. We use it as part of the face blur feature, and our media editing tool set and that was something that was requested in the summer of 2020. People wanted to post photos but didn't want to reveal people's identities. And so there is a you know, it's not actually that good. So you have to kind of go in and manually fix it. But it's an on device model that helps detect faces in crowd photos and automatically blur them so that then when you share them on social media, you're not revealing people's intimate biometric data to say clear view, right. So that's the one. Okay,
I'm glad I asked you, but
I think here's the thing, like, yeah, that's a great use of AI and doesn't that just, you know, disabuse us of all this negativity I've been throwing out from the stage. Sure, if that were the only market for facial recognition, but let's be clear, the economic incentives that drive the very expensive process of developing and deploying facial recognition technology would never let that be the only use so I'd rather live in a world with no facial recognition given what the economic model for facial recognition and companies like ClearView are, then and you know, just not have that feature. And you have to sort of use your thumbs to blur all the faces. Then live in a world where we yeah, we can use that. But ultimately, facial recognition is a tool of mass surveillance that is incredibly invasive.
Gotcha. Yeah, we don't we don't have too much time left I wanted to get to to quantum resilience in in your in your encryption. Can you just tell us Are we good there when the quantum mode decryption comes around?
Well, signal just became the first major messenger to to deploy in production post quantum resilience for a type of attack that we're concerned about called Harvest now decrypt later. So you know, that's pretty cool. And I want to congratulate Aaron and Rolf and the team who got that done. Why that's important is in a future where sufficiently powerful quantum computers do exist, which is still a bit of a hazy timeline, but we need to take seriously if we want to be serious about privacy. It could be possible for people who you know, collect the encrypted blobs of nothing that are signal data now to then deploy a quantum computer which would be able to crack that encryption. So with the post quantum update that we've done, that would no longer be possible there was you know, and you can read our blog post if you want to get into the details. And then if you want to really get into the details, there's a technical white paper which kind of goes through the specification of how we did that. So that's the first step of quantum resistance. And we're really glad it's out there because we want to make sure people who use Signal Private now and private in the future.
Yeah, that just rolled out so you can go wild now. No,
it's happening. I mean, if you updated the latest versions, it's happening. Okay, so
just a last bit of audience participation here. We did this this morning. But I want to do it with this this larger, you know, and number here, can you raise your hand if you have signal installed on one of your devices? Yeah, that's great. And now please keep your head while you put your hand back up. If you've ever donated to signal people, there's a few people well, we here we go. Kiersten. So, this is obviously signals importance can't be overstated. In so many departments like in the UK, you're fighting these, the law to just basically make signal impossible to own or use. What what is the biggest threat to signal and what can we do to help in 47 seconds? Well, we are
facing a global threat to digital privacy online. Full stop. That is taking the form of an emotional appeal to undermine end to end encryption in the name of protecting children. This is not a new trope. This is not a new appeal, but it has taken on really paucity in the last year or so. And there are a number of reasons for that and we are you know, that's existential for us. And I don't think there's a chance of a livable world if everything we say and do and every communication we have is surveilled by a handful of monopolies in Silicon Valley and the governments that they collaborate with.
So donate and can we want to do you want to do a
Oh, yeah, yeah. I'm gonna, this is for the team. So like, just give me
a look at the house likes to get a little, just a little quick picture for for Meredith. Thank you. All right. Well, thank you very much for joining us and thank you for listening. Thank you.