On Tuesday, May 5, Boston Review convened a panel of three prominent writers—Kevin T. Baker, Sophia Goodfriend, and Benjamin Recht—to discuss the way AI is changing the way individuals, institutions, and governments make decisions and the consequences for politics, war, and social life in general.
The following is a transcript of the event, moderated by contributing editor Lily Hu. It has not been fully edited and may contain errors.
Lily Hu:In the past few years, we’ve seen so many horrifying plot lines developing globally. Israel’s relentless and brutal war on Palestine; the AI rush and the associated rush to build data centers, ensure trade flows, secure access to key minerals and materials; the ever greater, ever-present risks involved in global migration; and now, of course, the war in Iran. We can now begin to see very clearly how these plot lines are being woven together. New technologies, and perhaps even more, the unprecedented levels of hype around those new technologies, are playing a transformative role in how our activities of war and surveillance are being carried out, and in the public discourse about these activities, how systems of tracking and shooting and killing are being pitched and sold and justified to the public and to the military by companies like Palantir. We’re hearing so much about more precise, efficient, smart AI-assisted warfare.
I want to start with that new gloss that’s being given to the usual campaigns of war and military force. These newfangled technologies grab our attention and they build themselves as making war or border protection more precise, more efficient, and it’s implied less costly in terms of human lives, but of course, as so much of your work discloses, this is not so. It’s still also human choices in the end that are making decisions to surveil, target, kill, and maim. I want to start by asking everybody, how do we remain clear-eyed about what’s going on right now in this new AI-assisted realm of military technologies? What’s new and what’s old? AI is changing so much of how war is being conducted, it’s changing so much about how social and political decisions are being made more broadly. How do we sort out the evergreen truths that are important for us to not lose sight of about these particular technologies, while also seeing what’s really distinctive?
Sophia Goodfriend:Thanks, Lily, and thanks for having us. I’m excited for the conversation. You’re absolutely right. It’s important to be specific and clear-eyed about how precisely AI is changing warfare, armed conflict, and the various atrocities that you’ve named and that frame our conversation today. I know that the other speakers here who have written extensively and have training in technical backgrounds can speak more specifically to this. But I would like to start out by clarifying first and foremost that AI is not really a coherent technology. It’s really a diverse set of tools from machine learning algorithms to large language models that are capable of automating tasks that were once carried out by humans. So when we talk about AI, we should really be talking about automation. And when we talk about AI warfare, we should be talking about how militaries are automating warfare and soldiering.
We’re seeing that today most strikingly amidst the U.S.-Israeli bombardment of Iran in the wake of two years of war in Gaza. Alongside all that’s happened in Ukraine as well, we’re really talking about how automation and artificial intelligence are transforming targeting on the battlefield. And that unfolds through a host of different technologies, from object recognition systems that are used to cull through reams of satellite imagery and inform anomaly detection systems that can alert a targeting cell in a military of when and where to strike. That can be facial recognition algorithms that help militaries compile lists of potential militants that they can determine are valid military targets and decide to assassinate using drones, or other kinds of guided munitions. It can be recommendation algorithms that can speed up the pace through which intelligence analysts can decide who or what constitutes a valid military target. And so that’s really the set of automated systems that are transforming much of warfare today, and I think it’s quite important to be specific about that. And when you’re specific about that, you can also be specific, again, about the kinds of technical limitations that these systems run up against that contradict the claims of both the private companies that are pushing these technologies, as well as the militaries that are using them.
For example, the anomaly detection systems within something like Maven Smart System manufactured by Palantir have a really, really high error rate when it’s used in different terrain that doesn’t match the terrain with which the satellite images that inform it were trained on. Or the automated translation tools that the Israeli military relies on to automatically translate reams of telecommunication data taken from Palestinians living in the West Bank or Gaza are known to have very high error rates and mistranslate words constantly, which can inform other kinds of errors and limitations throughout the larger kill chain. So again, when you’re specific about what kinds of algorithms and what kinds of technologies and what kinds of systems are being used by militaries, then you can get into the weeds of how these systems work and how they don’t work. I think that’s important, and I’m sure others can speak more to that as well, but I say that because when you roll out a host of systems that are used to automate tasks once carried out by humans, what’s new is that they do speed up the pace, the tempo, and the scale with which militaries can act. And we’re seeing that in Iran, we’ve seen that as well in Gaza.
In Gaza, the Israeli military integrated a host of AI-assisted technologies into its kill chain, and that allowed it to strike at the height of its aerial bombardment of the Strip once every two minutes. Fast forward two years, and you see that Israel and the United States were striking once every seventy-seven seconds in Iran. Just to sit with the pace at which these technologies allow militaries to strike and kill on the battlefield is also quite important. So that’s something that’s new, to go back to your first question. As you said, and as I’d love to hear others talk about as well, these technologies really subtend old dreams of warfare, old dreams of domination, old dreams of power.
The first, I think most importantly, is the fallacy that technological supremacy will deliver military victory and lasting security on the battlefield. It’s an old dream that militaries, Western militaries in particular, have offered, that achieving total dominance over an enemy through air power alone will shore up military victory. We’ve seen that throughout the 20th century, throughout World War II and Vietnam, in places like Yugoslavia, and now Iran, with the kinds of strategic failures that have played out over the last few months. And we see through that history that air power alone could never allow a military to achieve its strategic aims in warfare. And that idea has endured and is so alluring to militaries because it also comes with the idea that you could attain victory without sacrificing your own soldiers in war. And you can wage war without having to mobilize both the popular support and the political will of your constituencies. I think that’s an important old and enduring fantasy, and we can talk more about how that’s played out over the last few months as well.
Another old theme that tangles contemporary hype around AI and warfare is also the dream that automation is a technology of control. I think that’s something that we can see quite tangibly. Kevin, I think you cited David Noble in in your piece in The Guardian about how automation is first and foremost this kind of managerial tool, a tool that shores up the power of CEOs, of politicians, by taking the discretion and power away from workers and placing it into the hands of a smaller number of people. I think we can see that quite tangibly in how wars are waged and how militaries are embracing automation, particularly when it comes to targeting on the battlefield.
In Israel, in the context in which I’ve done my research, in the 2010s, as intelligence units were integrating a host of AI systems into targeting to speed up their operations, you had intelligence heads say that they dreamed that in a few years, 80 percent of the tasks that were once carried out by human analysts would be carried out by automated systems. That dream also chimed with quite specific aims that tangled with the dreams of right-wing politicians and the military commanders they worked with; of annexation, of displacement, of population transfer. Again, you can see how AI as a tool of managerial control also chimes with the specific political aims of militaries and governments that are deploying these systems, how they shore up the authoritarian aims of militaries and governments by again, limiting the amount of discretion, the amount of responsibility and agency that other people bound up in war machines or in governments, or wherever you’re seeing these technologies deployed, and placing it in the hands of a smaller group of people overseeing a larger system. I think that’s a through line we can draw through earlier experiments at automation and how militaries are today using these technologies.
Hu:One of the things that, Sophia and Kevin, I think you both touched on in your recent writings, is this emphasis on how these tools make claims to efficiency and precision and automation and control. But they are actually extremely faulty. They’re not as smart as they’re sold, and these errors often lead to even more destruction, despite claims that the war is going to be less costly, or is going to be more rational. So there’s, on the one hand, a set of critiques that’s focusing on this ideological debunking about these benchmarks of precision and efficiency and how they’re really not so, they’re actually extremely inaccurate; look at these actual cases, due to not updating a map, not updating an address.
On the other hand, there’s this worry that a line of critique that focuses on accuracy, the extent to which they’re meeting these marks that have been set up by this master narrative of what justifies the deployment of these tools, plays right into the hand of this logic of technical rationality, where the primary questions that we’re asking about war are, how precise is your targeting? Or, how accurate is your facial recognition technology? Or, how many mistakes did your system make? I feel like one thing that Ben’s book was really good at bringing out to me was the extent to which the problem is not simply that the tools or the systems are failing on their own terms, that they’re not as accurate, or they’re not as efficient, but that the terms themselves are deeply problematic. The ubiquity of technical rationality itself is problematic, and the extent to which we continue to measure up humans or tools to those particular metrics is sort of a deeper problem than falling short of the standards.
I want to ask, how do we fight both fronts at the same time? How do we both beat back the ideological hype about precision and efficiency, emphasizing that we’ve been ushered into this new age of hyper-rational warfare, and at the same time, reject those very goalposts of efficiency, accuracy, and technical rationality? Ben, do you have a thought about that, since I know your book focuses so much on what are these metrics to begin with?
Benjamin Recht:They all come out of experiences during the Second World War and the desire to make things more administrative, or the view that somehow some of the greatest successes of the Allied effort were in the deployment of smarter administration. I think after the war, there was this push to think about, how can you do that even better? We had all these engineers and mathematicians conscripted to work in these planning offices during World War II, and afterwards they just came up with this push for, like, let’s see how we could actually design that to make that more streamlined. You would hear a lot of frustrations from these sorts of people that the decision-making of the war was too ad hoc, and we would have been better off had we been able to mathematize our planning then. I mean, so much of the tooling behind everything that we’re using today in computing was set up to build systems to automate decision-making, to make things less ad hoc, to decouple means from ends. Everything in optimization and game theory starts there.
It’s funny, because the logic doesn’t change. The technology changes, but the argument stays the same. Probably the most tragic and obvious failure was Vietnam, where they were very incredibly technocratic. I mean, they didn’t have computers to the same extent, but the fun part is that you could do a lot of these computations with tables. You don’t necessarily need to have the actual machine there to calculate the logic of these spreadsheets. It’s interesting that the story doesn’t change, and that the kind of planning that would go into a lot of that campaign in Vietnam doesn’t sound that different from a lot of the logic behind the two Iraq wars, and isn’t too different than . . . I mean, it’s no different than what we’re doing now, although the technology is getting more sophisticated and invisible.
Hu:Kevin, I want to turn it to you. How do you think we can fold in the critique of the new old warfare with the broader critique of the old old warfare, which is the very premises of war at all?
Kevin T. Baker:You mentioned earlier that technology criticism faces a kind of strategic dilemma. Do we play into this conversation about precision and accuracy, and all of the ways these firms sell this technology? I think that’s a dilemma for a certain kind of tech criticism, but I think it’s one that we need to move beyond, because it locates all of the action within the technology itself, and lets the technology itself guide the conversation. What’s important to me about technologies like the Maven Smart System is the context in which they’re embedded in. The war in Iran and the war in Gaza are notable for, first and foremost, a complete indifference towards civilian casualties, a complete indifference towards what used to be called, euphemistically, ‘collateral damage.’ So, that’s an important calculus that we need to take into consideration. Another one is, these casualties are very straightforward consequences of doing warfare in a dense urban environment. They’re not something any kind of technology is going to be able to eradicate or displace.
The other thing is, these systems are often precise in hitting the target that is requested, but that doesn’t really get into the question of target selection to begin with. And all of these are essentially political questions and not technical questions. I think when we focus too tightly on the ways that technology is embedded within these politics, we lose track of what is still political, where we as people not in government or within these firms have an ability to change the course of history. I think a lot of this needs to take place not in the questions of the fundamentals of the technology, in the internals, but in much more straightforward political conversations about whether this war is legal, whether it’s moral, whether we should be doing it to begin with. This was one of my frustrations early in this iteration of the Iran war, where conversations about Claude, and Claude supposedly bombing a site, tended to displace questions about the war in broader terms.
Goodfriend:I’ll just echo what you’ve both said, and to Kevin’s last point especially, I think we saw it not only in Iran with the hype around Claude AI and large language models there, but also in how Israel’s war in Gaza was discussed, too. When all this reporting came out that colleagues of mine at +972 Magazine did about the use of AI in targeting there, the conversation became about this dystopian set of technologies which were driving this unprecedented rate of civilian casualties, when in actuality, what was driving up the death toll was concerted decisions made at the upper echelon of Israel’s government to say, yes, our laws of war, our rules of engagement will sanction killing 100 civilians in an aerial assassination that’s targeting an upper-level Hamas commander. Or, we’ll start targeting not only proven mid-level militants of Hamas and Islamic Jihad, but people at the administrative wing, and not only that, we’ll kill them when they’re inside their family homes. These are all concerted human decisions that are driving how the set of technologies that gets taken up as the driving narrative force in both critiques and celebrations of how militaries are waging war, and that occludes these quite concrete and quite human decisions that should be the subject of debate.
Hu:I think I want to shift to talk about some of those humans, but maybe some of the humans that, this time around, aren’t as visible as in the last round of big tech criticism that we had about ten years ago, in particular, the new collaboration of Silicon Valley and what labor has to do with that story. When I think back to the the last big round of conversations that a lot of these tech companies were dealing with, like public outcry about bias and accountability and responsibility—social responsibility, to use corporate talk—issues about war seemed like a third rail. Kevin, you talked about Project Maven. That was an incredible rallying point for so many workers at Google, and that managed to lead to a rejection of the project. Obviously, it picked back up, as these things always do. But not there, anyway. There was a kind of broad-based worker dissent about cooperating on these projects, and even it seemed that management was sympathetic to those concerns, or at least acceded to them in notable cases.
I think that went along with, in 2016, what seemed like a broader industry-wide openness to that type of accountability as bearing on fundamental business choices or design choices in the products that they were selling. We saw this in how many of these firms navigated DEI issues. We saw how they were talking about democracy, creating a hospitable speech environment for liberal democracy. And it seems like this time around, with this rightward turn, it’s not just Palantir, but, the formerly better guys, as it were, of Facebook/Meta, Google.
So how do we take that? Do you take that, Kevin, as a sign that ten years ago it was all smoke and mirrors and lip service for that particular political moment? Or do you think that things are genuinely different ten years later in terms of the broader labor force at these companies and their militancy on these issues, or in terms of these companies, C-suites or boardrooms, or other kind of broader industry or economic conditions?
Baker:I think there are a couple of different things to disentangle, but I think I would say that the workers during the Maven walkout almost a decade ago were navigating a very different employer environment. I think firms in that period were much more sensitive to reputational risk, and this is something workers were able to leverage and exploit to a certain extent to extract demands. This is something we also saw in higher ed organizing. I was building a union at the time, and we were able to extract meaningful concessions from our employer without having a recognized union yet just by exploiting this gap between the corporation or organization’s image, and what it actually did in the world.
I think a lot of things have changed, though, in the interim, that make these strategies harder to use. One of them is, I think I would push back on the idea that collaboration between the military and Silicon Valley is new. DOD contracts and DARPA contracts have always been a major part of the bread and butter of these companies’ business models. I think what has changed is how they imagine the government buyer for these. They’re speaking to a different audience. I think even into the first Trump administration, these businesses saw themselves as selling their product to a permanent liberal governance class. This was the impetus behind a lot of the fairness discourse at the time. This sort of deal gradually ended up collapsing over the Biden administration, and it’s a much different reality now.
But, just today, Google DeepMind workers in the UK voted 98 percent to form a union, and they’re making demands of their employers to end military AI use, restore the weapons pledge that was instituted during the Maven walkouts, and other demands. So I think there’s a recognition that awareness-raising strategies aren’t going to work anymore and it’s going to take more traditional labor militancy in these firms to make a difference. But I think it’s also going to take much more direct political confrontation outside of the labor process within these firms. These are ultimately national political questions, and we need to treat them with that.
Recht:I feel like the most important point is that they have far less leverage today than they did in 2018. In fact, perhaps they should have tried to extract more. I guess we always see in hindsight. It’s a complicated issue because these workers are very comfortable workers, and you never know how far anyone wants to really push. At least in 2018, they wanted to make sure they were heard. I think some people much more sincerely than others wanted to make sure they were heard, but everybody else wanted a certain kind of affirmation, and they still wanted that affirmation to take home a very large salary, which is particularly complicated. It’s only exacerbated now in terms of their salary, and their strength is far diminished, partially because so many of the owners of these companies have decided that they’re all in on Trump, and partially because all the owners of these companies have decided they don’t need these workers anymore. It will be interesting to see this particular movement from, very importantly, the London office of Google DeepMind, and what exactly they’re able to do. I’m not disparaging their beliefs. I think that they’re trying to do the right thing. I just think it’s . . . I’d be curious to see how effective they can be from where they are.
Hu:Ben, I want to ask you a different kind of question about perhaps the biggest shift in tech, and arguably in the entire economy, since it seems as though at least the entire U.S. economy is just AI these days, and what that even means is increasingly unclear. Of course, I’m talking about advent of so-called generative AI LLMs, which seem to have entirely eclipsed, in terms of language and diction, machine learning and algorithms, even though in many cases, those terms are more apt than just saying AI, or definitely than saying Claude. Do you see the shift to AI systems like Claude or ChatGPT as suggesting a shift away from the previous models of machine rationality that you talk about in your book, which are based in optimization?
You emphasized how machine optimization is really not like human reasoning. I love the case that you talk about with chess. I mean, sure, machines have players better than a human chess player now, but if you try to get a machine chess player to play like a human, they’re way worse, right? So does this shift to LLMs suggest a shift toward something more like human expertise as opposed to optimization? Because it seems like the allure of these chatbots is that they seem like us, and they’re trained on us. That’s a far cry from linear programming or game theory or behavioral economics, null hypothesis testing, this sort of bread and butter of the previous iterations of algorithms and machine learning. Do you sense that this is a fundamentally different type of rationality that’s encoded in these systems? And as a follow-up to that, what does the apparent willingness of so many people to turn over their lives to these systems mean?
Recht:I think there’s one thing that’s always tricky, and I think this is what Kevin’s piece in The Guardian was really careful and very important to make this argument, is that a lot of underlying logistics systems and decision-making systems are still the old ones that we’ve been building since World War II, they look very much the same. And it’s easier for us to believe, because people are talking about how smart these new technologies are, that it was actually this new stuff, generative AI, that’s making these decisions, as opposed to the old stuff, which maybe we think of as passé and not just out of style, but technologically backwards. But I think that’s not the case, and hopefully Kevin will speak to that in a second.
But before that, I think your question is a great one, as to, if you have these things generating language, are they operating with an internal system that’s different from our boring optimization system? I’m not sure. I mean, I think that the optimization mindset has certainly captured everybody in Silicon Valley, and everybody in Silicon Valley is using their agents and what have you to make themselves feel like they’re being more optimal, in whatever axis of optimality they want to go on. It’s certainly still the case that people are trying to be more like these old ideal computers, now just with a sycophantic friend that they’ve created in their head, echoing back what they need to hear. And certainly, it’s also the case that the technology that the LLMs are built on is a pretty natural extension of what people have been doing since that same time period. In our computer age, we just don’t have that many clever ideas other than making these things faster and smaller and just seeing what plays out at those extremes. I do feel like a lot of where we are now is at this logical endpoint, which people have been striving for, and now we’re just confused by what we do with it.
It is funny, right, that these things talk back in language now. And that language can be very . . . I mean, it’s been harnessed so hard, so it can’t really be that creative, but it could feel like it’s speaking in poetry rather than in logic. But that poetry is just a logical synthesis of the internet, spit back at us in the most probable way for whatever end it’s trying to get at. So, I don’t know, I love the question, I don’t think I have a good answer for it yet. I’m still trying to think through all the ramifications of getting to this endpoint where we’re trying to write programs and trying to actually push our goals in natural language, when that almost seems to be the antithesis of what we were supposed to set out with. We were supposed to be disciplined in how we stated things, and by stating things in math, that was going to allow us to be more clear-eyed about what our endpoints were. That never really was the case, and I think we’ll see where we end up with, where we now just say, tell computer to do this in natural prose and hope for the best. I don’t think we’ve seen the conclusion of that part of the story just yet.
I would love to hear a little bit, Kevin, if you agree with my assessment of . . . I mean, I know you have a lot more to say about this, but I would love to hear a little bit more about your feeling behind the logic of Maven versus the logic of this Claude system, and how you see them fitting together through your research.
Baker:I think one thing that this generation, I hate the term AI, but what we are calling AI now, one thing that it does that previous generations of AI struggled to do, claimed to do, but never really actually was able to even approach was a labor of commensuration, of being able to translate from one form to another. And this was work that was traditionally filled by clerks, bureaucrats, secretaries. It was work that the computer revolution claimed to have displaced, did displace formally, but not by actually doing the work. Very little is new, I think, about the way this technology is being deployed, but I think that is new. I have no idea if that actually relates to what you asked, but it’ll have to do.
Recht:I think it’s a great point. No, I think I was just asking about, why is it easier for people to believe that the bombing decisions are being made by Claude, and not by whatever—or also, it would be helpful to explain, how exactly does Maven make decisions?
Baker:Well, it’s kind of a co-production. You’re basically running various stages of the targeting decision through a workflow, and this is partially automated. And this dynamic itself is actually not new. It goes back to the very origins of defense computing, something like the semi-automated ground environment, which was the first real-time computer system that hooked up radars to computers and attempted to track incoming bombing runs. There’s always been a kind of cybernetic synthesis between human and machine, and I think the human part of it tends to fall away discursively. I think partially because of how these systems are funded and sold, but partially because critique itself buys into this to heighten the urgency of the discourse. It’s not the same old problem; it’s something new and scary. And it is new and scary. It is a pitfall though, and it does do something to our ability to hold people accountable. At the end of the day, I think we need to say, you can’t say you’re just following Maven’s orders, you’re not following Claude’s orders. We need to get back to some basics on this.
Hu:Sophia, do you have any thoughts about accountability? I feel like there was a theme in both your and Kevin’s pieces about the return to responsibility. How do we recenter that there is, in fact, responsibility at the end of the day for these decisions, even in complex, technical, automated systems. What is the political purchase we can get with the language of responsibility, and do you think there’s another alternative focal point of critique here?
Goodfriend:I think responsibility is quite key in how we talk about how these technologies are being used and deployed, especially by militaries. I think that they offer militaries today a kind of ideological function of denying responsibility, and we see that both in how atrocities like the Minab bombing were discussed by the Pentagon, by political leaders, and by the media in ways that sidestepped questions of responsibility and accountability, and who precisely was in charge and could be held accountable when it comes to discussing the killing of over 100 schoolchildren, for example. So, I think it’s quite important that we center it, and that’s the case for a few reasons.
Also, there’s the allure of these technologies, for those who use them, of denying their own responsibility and complicity in their own actions. I see that in interviews that I do with people who’ve used these technologies in targeting as well. When they’re bound up in this massive war machine, it becomes much easier to offload any sort of complicity in their own role in atrocities onto a statistical mechanism like a targeting algorithm, and to not say, this was a decision that I myself actually made. I think placing that question of responsibility at the center of our critique also affirms that these are, once again, human decisions, that humans are bound up in them, and that there is an exorbitant human cost to how these technologies are used and deployed. These technical systems seem really advanced and seem really complicated and confusing, and most of the people who are writing about them also don’t really know how they work, and most of the military leaders talking them up don’t really know how they work, so it’s easy to fetishize them and give them this magical power, when in reality, again, it’s humans who are using them and humans who are bearing the quite profound consequences of their effects, be it in the military or in other sectors. So, I think it’s a discursive imperative to center that in how we write and think about them and talk about them, but also should be institutionalized in other ways as well.
Hu:I think that the focus on Claude is actually a kind of interesting expression of this desire to hold something responsible. The fact that Claude is so easily anthropomorphized, I think, maybe partially explains it, in addition to it obviously being much more shiny at the moment, but makes it more liable somehow to absorb responsibility. As compared with Maven, which seems not suitable to taking accountability. It just seems like an operating system or something. People say this now with people’s uses of ChatGPT, they’ll say, you wrote it with ChatGPT, but no one would say you wrote it with Google Docs, or you wrote it with Mac OS 10. There is this anthropomorphization of the models that I think is responsive to the OS, the need for something to have done it, and it seems like it’s really hard for us to think that it was just a machine, and so there is this allure, which I think is kind of interesting. We need to be not playing into that particular type of anthropomorphizing, and there’s actually real humans, decisions, policies, politics here, and that that’s the actual correct locus of cries for there to be accountability.
I want to now zoom out and ask everybody their thoughts on where we are now, the big question. We are obviously in heretofore unseen levels of hype about technology, and we’ve been told again and again by both the critics and the boosters that we’re on the brink of something truly transformative. Either transformative of society, of our economy, of our own personal lives, whether that means luxury communism and the end of all work or catastrophic inequality and a return to a feudal system of bondage for most people. It seems like everybody is saying the world is going to look very vastly different in just a few years. What do you think the popular discourses about AI are getting wrong? And what should we be talking about instead?
Recht:I don’t like making prognostications, because I never know. I think denying that language models are a revolutionary technology is ridiculous. You can’t do that at this point. They certainly are transformative for a lot of things, but what’s actually going to happen? I have no idea. What I find very frustrating is that you have economists who are willing to toady up to the likes of Silicon Valley CEOs and say what they want them to say, and somehow that’s supposed to make it more true, that as long as you can get a big enough group to agree, then so it is. But I don’t necessarily see an end of work. I don’t know, maybe I’m just being too optimistic, but I think if people want to work because they want to make a living, I think they’re going to be okay. I don’t think that technology is doing what a lot of the boosters want to claim is doing. I don’t think the technology is useless as some of the other people are claiming as well. I just think it’s different. It’s certainly providing a new way to think through or to approach problems, especially in software engineering, than we used to have.
This is not the topic of conversation at all, but I certainly think that the place where it’s had the most impact, and will continue to have the most impact, is in education. Kevin maybe could comment a little bit more about this, because I saw him note this on social media today. I think that it does seem to be a weird endpoint of being so obsessed with a certain kind of commodification of the degree and a standardization of testing for everything and making everything into education technology, that at this point people were like, if you can tell me all of the metrics I have to cross, I can certainly build a machine that will hit all the metrics. And we’ve built it. And what we do with that is going to require us to have a big evaluation of what we do in our line of work.
Baker:There was a scholar of social informatics who’s not widely read anymore, but probably should be, called Rob Kling, and he viewed the things that went on with computerization and the reorganization of society as expressions of technological, social movements, people who use a technology rhetorically, and sometimes practically, to achieve certain political or social ends. And I think what we are seeing in the transformations wrought by AI are less expressions of the technology itself and more the expression of a social movement among the rich and the powerful. We’re generally bad at seeing these kinds of social formations. We either view them conspiratorially, or we don’t see them at all. But if you look at the movements among CEOs, among deans, they’re saying pretty much the same thing that ed reformers were saying a decade ago about laptops, about Chromebooks, about whatever. They have a social goal, and they’re using the rhetoric around technology to achieve those social goals.
I agree with most of what Ben said about the fundamentals of this technology. But I think the thing that I will add is the one wrinkle in the knot of the work replacement theory that these things are going to completely displace us is that LLMs are kind of unruly in the same way that human workers are, and they don’t always generate the expected outputs, and I think that will eventually drive management absolutely crazy. It looks good now, but I don’t think it’s a winning bargain for them in the long term.
Recht:This week I have seen a lot of stories of managers being frustrated with the quality of the software these things have been shipping and the maintenance of those code bases. So, just like everything in AI, it might happen sooner than we think.
Goodfriend:I don’t have too much to add. I think that both Ben and Kevin have been good about emphasizing both how these technologies do and don’t work, and if we can really take these promises of labor replacement and everything else seriously. I would just emphasize that all kind of promises of automations, magical effects can be disrupted if you just look at the sprawling and quite material supply chains they hinge upon, when it comes to the use of automation and militaries.
Israel’s military, for years, and that’s just the context in which I did my research, for years would promise that the automation of its occupation would give form to this frictionless military rule where you wouldn’t have soldiers being deployed to maintain occupation in places like the West Bank. But in reality, each successive automated technology being used hinged on quite embodied forms of military policing, of soldiering, and that’s not something that’s unique to a military context. It’s also how companies like OpenAI build up their products by using undervalued workers overseas to annotate data. You see these kinds of similar processes of not labor replacement, but labor displacement, disrupt these quite utopian claims of CEOs or politicians or generals at every step of the way. Being attentive to the humans bound up in that supply chain, the material forces that are subtending the AI industry wherever it’s taking shape is, I think, important today as we continue to hear the same old story about what these technologies will do, for better or for worse.
Hu:I’m going to turn to a few audience member questions. A couple of people are asking about what we can do in our capacity as private citizens or consumers about Silicon Valley and our attack overlords. One of the audience members asks, is it possible? Would toppling or capturing Silicon Valley make it possible for others to make these technologies work for humanity instead of profit and war?
One nearby question I had about this is, there’s a consumer-facing aspect of these tools and these companies. That’s my fun therapist in my pocket, or that’s my homework helper. And then there is the same technology or company engaged in clearly sinister military projects, imperialist endeavors, and it seems like there’s a disconnect with many people in recognizing this fact. Is that an opportunity? How should individual private citizens, consumers, approach this industry? What is actually concretely possible? Do you have any political stratagem to offer us?
Recht:I feel like it’s a great question, and it’s a question we’ve been asking for a decade, and no one has an answer to. It’s not just that we’re frustrated now. We referenced being frustrated with Project Maven in 2018, and it’s so interesting how that just moved out of the public eye into the shadier realm. You moved from Google to Palantir, and it happens anyway. Someone else is always willing to pick up that contract. Google is just as entrenched as it was in 2018. As Kevin pointed out, they’re less afraid of public opinion than they were in 2018. They’ve embraced a politics that’s certainly more right of center than it was in 2018. It would be nice to have some hope, right? It would be nice to have a path.
Hu:I will say, when talking to people, there’s always the refrain, ‘people hate this stuff.’ There’s always that refrain, and one wonders, A, the extent to which that’s true, and B, how to harness that type of anger or that type of ‘nobody wants this’ feeling. We have the seven richest people on the earth determining the future for us, for which there is no off-ramp. How is that harnessed? Obviously there’s been organized work and activism against the building of data centers, but I’m wondering what other strategic points of direct actions are possible.
Baker:I don’t have good answers. What I will say is that since the beginning of the pandemic, the ability of the broad left to organize has been broadly diminished. Political organization within a left framework usually relies on deeply embedded networks of social trust and friendship, and I think a lot of that has died out. And we have a situation both with the war in Iran and with these companies where public opinion is massively against them. This war is unbelievably unpopular. These firms are unbelievably unpopular. What we lack is, I think, an organizational capacity to take them down, and there’s no easy way to build this. It involves talking to your neighbors, your coworkers, and getting together, rebuilding this capacity. I don’t think consumer boycotts are going to do the job. These contracts with the Department of War are too lucrative. It’s going to take something else. It’s going to take old-fashioned politics, I think.
Recht:I guess the one thing I’ll speak to, and the reason why I’m kind of bummed out today about it, is I see a lot of young, bright people who graduate and then just go into this anyway, who I think, if it had been 2018, would have had a very different conversation than, ‘Anthropic’s the least bad option.’ I’m speaking from my own place as a CS professor. I see where my students are going. They’re buying into this. So it’s a little tricky. I feel like I don’t know if the four of us are a little old-school here, but I’m not sure that I have a good answer. I think that the one interesting thing is, we have been seeing a weird grassroots movement against the data center, which I find fascinating. I’m not sure what to make of that, but that’s one little place where people seem to be really pushing back. I don’t yet know what to make of that, but I find that to be a very interesting development.
Goodfriend:To echo what others have said, I definitely don’t have the answer to that very important and good question. But I also wanted to ask Ben or Kevin earlier when we were talking about the news that came today that the workers at the Google DeepMind office in London had unionized. Amidst all this talk of labor replacement, especially among white-collar developers and people who make up that union, the shifting nature of what it means to be a worker within these companies is interesting when it comes to also talking about what kinds of pressure from the inside would have an effect. When this question is asked in other events that I’ve been in, people cite these longer histories of tech workers revolting or drawing attention to the really atrocious uses of the product of their labor, and that kind of worker paradigm really being pivotal. I’m just wondering if that history is relevant, or if we’re facing something different, given the changing nature of labor within these corporations, of the human worker in general, and what other kinds of paradigms we need for politics and the like. But I’m doing the unfair thing of asking a question to the question that was asked of me, so you don’t have to answer it if you don’t want to.
Recht:But it’s such a good question, though. Especially given the companies’ position that their whole goal is to end white-collar work. What a funny thing to be working at a company whose goal is to put you out of your job, but then to be dangled the promise of being able to retire early. So maybe you beat everybody else. It’s such a funny mindset. It’s a very peculiar thing, being a computer science professor right now, I’ll just say that.
Baker:I don’t have good answers to this either, but I think one place to look would be the long history of secretarial labor in the ’70s and ’80s, and attempts to replace this labor with computation. I think there are very clear resonances between now and then. This is a multi-generational project to replace bureaucratic labor, replace clerical labor. It’s been the dream since the dawn of computing. We can look to the past, for maybe not answers, but some sort of reassurance that we are not in truly unprecedented times. People have fought back.
Hu:I remember I was in Berkeley last month, and I took these photos because I was confused about the irony that might have been expressed by the ads. I couldn’t tell whether they are ironic. I’m too old, I guess. And they said, ‘Most AI companies avoid saying they’ll automate jobs. We don’t.’ Okay. And it’s a $300,000 base plus competitive equity. And the next ad says, ‘Automate software engineering before someone else does.’ I think that this is a curious kind of nihilism. Maybe it’s like, well, I might as well cash in now. The irony of the ad seems to reflect that type of doomer mindset, as it were.
We have a historical question about whether you guys are familiar with a previous time in history when there’s been such an extreme mix where everyday people hate this particular technology and the elites love it. This audience member asks, was it also like this in the ’90s when the internet was getting popular? Are there any historical analogs for this combination of broadly unpopular tech embraced by elites?
Baker:I think there are, but they’re pretty similar to this one. I mean, the original techlash, I think—well, we’ve always been techlashing, but I think the original techlash was in the ’70s. There was massive, massive hatred of automated computer billing, of various ways that firms used computers in the 1970s. People getting water bills for multi-million dollars due to computer errors. There were stories every week about this kind of thing. I think it rhymes with a lot of what I’ve seen. You also see kind of similar questions or similar things going on with the electrification of work, most notably in like in a movie like Metropolis, or Chaplin films. But these were technologies that reorganized labor in very psychologically destructive ways. Telephone workers and various kinds of electronic workers in the early 20th century came down with severe mental illnesses due to the pace increases and the specific nature of their jobs. I think the hatred is something that I see a lot in history, but I don’t see a lot discussed.
Hu:We have another question about an AI summit that’s happening in June in New York City, sponsored by the McGovern Foundation, which is a philanthropic foundation, and the Aspen Institute, whatever that is technically. This audience member says it is basically a Big Tech institute. What do you all think about discussions about this being held under the sponsorship of these types of hosts? Do they really offer an independent perspective? Are they really an independent stakeholder, as it seems like they might suggest by being foundations and institutes and think tanks?
Baker:My opinion is this is the original form of slop. Things coming from outlets like the Aspen Institute or any of these kind of elite—I don’t even know what to call them. They just co-locate powerful people. Knowing nothing about what’s going on with this, I don’t have much hope that anything meaningful will come out of it. But I’m also being kind of reflexively dismissive.
Goodfriend:Yeah, I don’t have much to add. I would just say that in my experience of going to a lot of Big Tech-funded tech conferences—I mean, maybe Ben has a different perspective, but at Big Tech-funded tech conferences/military industrial sponsored AI summits, often the conversations that happen under the banner of critical takes on these technologies or ethics do just kind of sound like the slop come to life. There isn’t much grounded, deep reflection. But again, I think it’s also contingent on who’s on the panel, and how much they feel empowered to say.
Recht:Yeah, I think that’s 100% right. I think that one of the tricky parts with philanthropy is you can’t piss off the philanthropists. I do think that the other tricky thing is how much capture there’s been from Big Tech on not just these philanthropic organizations, but even government ones, through these philanthropy networks. We see a lot of that. So much crypto money has gone into now funding all the AI safety institutes who will show up at these meetings because they’re part of the crowd. It’s interesting how the money is shaping our discourse. Those people end up not just at these AI summits with the philanthropists, but at AI summits at the White House, too. That concerned me even more under the Democratic administration, which should have known better than under this current one, where whoever pays gets in.
Hu:Is there any mapping out of other types of counter-forces to these technologies or these firms? I mean, I hesitate to say, but there was maybe a promising alliance by neoconservatives and the religious right against many of these technologies—of course, not necessarily concerned with the particular war ventures that they’re involved in. What are the other sources of political power that could be opposed to this type of technology, especially in the context of military and war?
Recht:One of the things that hasn’t happened in this latest boom with large language models is a more grassroots ability to be able to do these things from your house. I think what ended up happening the last time, a decade ago, when we had new tools for image recognition, was pretty quickly hobbyists could actually build and deploy and share these things to a surprising degree. For a long time, Google or Facebook would claim they’re the only people who could run this technology because of scale, and then they were proven wrong. But we haven’t quite seen that yet with language models, and in fact, what we end up seeing is that the counterbalance tends to come from China, of all places. Their Chinese models are the ones that end up trying to push the frontier models, which makes this now a geopolitical crisis.
I feel like this is getting into some bad ’90s techno-optimism, but I do feel like one thing that’s missing here is the push to actually democratize the technology, to actually make these kinds of new AI tools doable by hobbyists. Relatedly, one thing we haven’t touched on is the way that this technology can be used in counterinsurgency movements, which I think it has been. I mean, the fact that you can make very cheap improvised flying explosive devices is something that could be used not just by giant governments, but by small people who need to fight back as well. I’d be curious to hear if Sophia and Kevin have seen much on that front recently or have thoughts along those lines.
Goodfriend:In terms of opposition, it can go in two separate ways. One, with the question you just posed, Ben, I think we’re seeing that happen more and more with the use of ad hoc different kinds of machine learning systems by insurgent groups, especially in the Middle East, when it comes to smaller drone systems equipped with various AI capabilities. If you read through defense newsletters that are sent out among militaries, that’s something that’s come up over and over again and sparks a lot of anxiety.
But I would shift the framing, too, because what I’ve seen a lot is also pushback from within militaries as these systems are being integrated into the pipeline and how they operate. I see that on two fronts, both among lower-level soldiers who are using these systems and become whistleblowers because they see the ethical effects of them on both the kind of operations that they are forced to be conscripted into, and also just what it does to their own capability of discretion and decision-making, and being really, really terrified by that. I find a small sliver of hope in that, precisely because the effects that these technologies have, not only on people who are living under aerial bombardment and warfare, but also on people who are forced to carry that out, are quite dehumanizing, and there is pushback on that level. People will frame it as disempowering, like, they have no power, they are cogs within this larger bureaucratic machine. It tugs at these older critiques of industrial-scale killing and industrial warfare that have been around for decades and decades. So in terms of opposition, I think that there are possibilities there just in how people who are conscripted into using them in really atrocious ways respond as well.
Baker:I do think if there if there is an opportunity for oppositional use of this technology, it’s probably related to hardening security within organizations. One of the reasons why these attacks on Iran were able to be successful is because of deep hacking of traffic cameras, other kinds of cameras, lots of surveillance. And this is something I think we’ve seen in the government response to Mythos, the upcoming Claude model. They’re afraid that their zero days will go away and they won’t be able to infiltrate computer networks in the same way that they previously were. In terms of the other part of the question, I don’t know.
Hu:Thank you all for this conversation. You’ve burdened us so much with your knowledge and expertise and political thinking, and I think that we all have a lot of work to do and a really open future. I think that there’s a lot of ways all this could go, and I’m hoping that there will be clear political opportunities and strategies moving forward.
Independent and nonprofit, Boston Review relies on reader funding. To support work like this, please donate here.
