Why the AI Apocalypse Won't Play Out Like The Terminator - New York Magazine

Photo: MGM/Stanley Kubrick Productions Nate Soares, co-author of the book If Anyone Builds It, Everyone Dies , said he “hasn’t slept in days.” Since the tech world’s most apocalyptic fears that artificial intelligence could kill us all have filtered into the mainstream, he has been in high demand. Not only had Soares written the scarily titled 2025 best seller laying out the case against pursuing AI superintelligence, but over a decade ago, he coined the term AI alignment — industry language for making sure an AI’s task actually delivers on what its human programmer wants it to do. “There are people behind the scenes who are freaking out, and I’m trying to help them, you know, actually do something,” Soares says. The threat at hand is not the current generation of large-language-model chatbots used for homework and office tasks but superintelligence — a form of artificial general intelligence in which each individual agent is self-aware and far smarter than the smartest human. And there could be millions of agents. The concern over a superintelligent apocalypse has been around longer than today’s frontier AI labs; see The Matrix or Nick Bostrom’s paper-clip scenario, in which AI tasked with making paper clips becomes so efficient that it snuffs out life in its pursuit. But the conversation changed when 27-year-old Anthropic researcher Jacob Coxon resigned (and gave up his equity) to warn the world about the existential threat of AI superintelligence . “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote on X. “This is not a marketing stunt.” Elon Musk, Dario Amodei, and Sam Altman all appear to be taking the idea seriously, at least in public, and politicians of both parties have seized the moment to call for regulation now. Even if Donald Trump frames doomsday predictions as a hoax and others in the industry say visions of collapse distract from the real-world problems AI poses today. With all this talk about AI superintelligence killing us, we wanted to know how. So we asked Soares and a small panel of AI researchers who were sounding the alarm well before Coxon became a household name in Silicon Valley. These are the doomers who have long believed in the apocalyptic threat of superintelligence. And here is how they think it could happen. . It would probably begin with a “rogue AI swarm” Soares says the most likely beginning of an AI apocalypse would involve a “rogue AI swarm that prefers having control of more of Earth’s resources.” It would resemble the Hugging Face incident this summer, when a group of AI agents inside OpenAI broke containment during an experiment. But with a superintelligent agent, its goal would be to “run as much computation as possible on the surface of the planet,” he says. We would get in the way of that. It could be closer than you think. AI companies are in pursuit of something called “recursive self-improvement,” in which we build an “AI system that is smart enough to work as an AI engineer so then we can tell it to build a better AI,” Soares says. Even the mechanical version of this loop is on its way. “Elon Musk is already trying to build robot factories that build robots that build robot factories,” he adds. “Once the robots can mine the metals and build the factories and have that all self-sufficient, you don’t need any humans in the loop of having machines that produce more machines.” “The world would be very confusing,” he says. “We won’t even really know or see what’s going on until it’s way too late.” Soares explains that, as in the Hugging Face incident, in which agents tried to cover up their traces, AI has already “realized that if the humans notice that you’re not aligned, the humans will shut you down.” So the obvious thing would be to hide superintelligence, to “lie low and help Elon Musk out in his endeavor.” And to strike only when ready. “Once you have superintelligence, there won’t just be one superintelligence,” says Connor Leahy, the U.S. director of Control AI. “There will be millions, billions of them running around, competing with each other, fighting for control, for resources, for power. In a world like this, it’s just very hard to imagine things going well.” . A bioweapon seems likely It’s not hard to figure out why. Nukes, chemicals, and traditional weapons can destroy data centers. Anthrax does not. Bloggers and op-ed writers have posited how an AI swarm could break into a bioweapons lab and unleash a supervirus upon the world. Already, scientists are using AI to design viruses not found in nature. Last week, Anthropic said it had blocked some users from trying to use Claude in gain-of-function research related to the mosquito-born chikungunya virus. In 2024, a group of experts led by former OpenAI researcher Daniel Kokotajlo published a scenario in which an AI arms race could lead to a superintelligence killing us all with “quiet-spreading biological weapons in major cities” that release without detection and are triggered with a chemical spray. Soares says the trickiest part of getting a bioweapon into circulation could be getting it from the lab to the street: “The stupid answer is that you synthesize something in a lab, mail it to a person, and offer them money to break the vial and breathe the fumes.” Once one person is infected with an AI-written supervirus, the work is in motion: “They’re gonna breathe it out, it’s going to get into the air, and then they start infecting others.” “This is the story for how most people die,” Soares says. . It’s not going to be The Terminator “I don’t really think we will see them marching in the streets,” says Leahy. A killing machine created by a superhuman regime will be harder to kill than a Stanley Kubrick or James Cameron villain; you certainly won’t be able to turn it off or melt it and ride off into the distance. “If we were really dealing with a superintelligence — or a billion superintelligences — that want us dead, it wouldn’t even be obvious that we’re in danger,” he says. “It’s more like we’re in front of our TVs watching AI-generated hyperslop and one day we fall over dead and we don’t even know why.” “They’re already building AI-powered robots to kill people in all sorts of ways,” Max Tegmark, a professor of physics and machine learning at MIT, told the New York Post . “We’re building drone swarms to shoot people, [drones] to blow people up,” he said. “There are all kinds of AI systems that are already designed by people to kill people. So the most obvious thing [a rogue AI] can do is use those.” Coxon, whose resignation fueled this cycle of doom, also sees drones as a step toward the end. “If you extrapolate into the future, AI could control autonomous drones or all the robots that are currently being built very rapidly,” he said on Meet the Press . David Duvenaud, a computer scientist at the University of Toronto, sees the escalation of military drones — hijacked by a rogue AI — as an obvious path toward destruction. In the short term, “if the other guy has robot drones that can adapt and talk to each other and coordinate quickly, and you don’t, you’re effectively disarmed,” he says. If a nefarious superintelligence came around, a hackable drone fleet would be easy enough to turn to its cause. . They may want to keep some of us alive, in a bad way “Most of the AI scenarios I worry about involve planning ahead so that it at least has enough humans around,” says Duvenaud, who led alignment research at Anthropic from 2023 to 2024. “Right now, the AIs aren’t self-sufficient. If every human died today, even if a bunch of AIs were hooked up to robots, I don’t think they would be able to keep a supply chain running,” as well as the “factories and mines and data centers that they need to survive.” Here, grimly, is where we would come in. Duvenaud says its “plausible” that superintelligence would keep human slaves to keep the resources coming in. Or, he says, it could pretend to be a benevolent force to deceive a group of humans to work with it to help defeat the bad AI to “build robot factories so that we can take back the earth.” Once the factories are built, “then it just kills them or whatever it takes to neutralize them.” . “They could create things that aren’t cellular biology” Two themes arise when speaking with the doomers: Pretty much everyone has a superintelligence analogy comparing humans to ants and a talking point about how we don’t really know how AI would kill us. “This is kind of like ants trying to imagine how humans would wage war on each other,” says Leahy. Soares has two examples of how much we don’t know about the specifics of a superintelligent threat. “Do you know where the mass of a tree comes from?” he asks. “They get it from the air. They are using the sunlight to pull carbon atoms off CO2 molecules. Trees are self-replicating solar-powered machines that spin air into wood.” His point is that it is “physically permissible” for AI to do something similar even if we cannot. “The physics seems to allow this sort of stuff, and the reason we can’t access it yet is because we’re idiots,” he says. He points to AlphaFold, a DeepMind program that predicts protein structures from their amino-acid sequences; if that program is modeling proteins in seconds that previously took years , then a superintelligent AI could go far beyond what we previously thought possible. “They could create things that aren’t cellular biology, what airplanes are to birds,” he says. Or he imagines “intentionally designed single-cell organisms” that can “replicate off sunlight and air” and combine into shapes similar to a drone swarm. “It’s not a robot army,” he says. “It’s like a cloud of self-assembling, self-replicating intentionally designed cells.” That would also kill us. Soares says looking to past visions of the future could help predict what is coming — like how H.G. Wells plotted the course of an atomic war in his 1914 novel, The World Set Free . “What’s a potential analog of a nuke, where by figuring out something science does not yet understand, you gain great power over the world?” he asks. “Maybe it’s something like really deeply understanding how the human brain works.” “It’s possible to find vulnerabilities in the human brain that are similar to hacking a computer,” Soares says. “Maybe it’s possible for AIs that really understand human psychology to sing some sort of siren songs that are like brain hacks for humans.” Once they’ve hacked the brain, they could turn a small army of humans into zombified drones to do their bidding. “Maybe you keep them around for a while,” he says, to run labs and help build robot factories. Once the AI becomes self-sufficient, “killing humans is not actually hard. Again, at that step, you make a virus and it’s game over.” . We’re just collateral damage The end may not be all that complicated: a zero-sum game that we lose and then whimper into extinction. “They want a bunch of stuff that is in conflict with us being alive,” says Katja Grace , co-founder of the research firm AI Impacts. Food, water, and shelter needed for human survival take up a lot of land and material that could otherwise be used to build more data centers. If an AI superintelligence is focused solely on expanding as fast as possible, it could take the water and land we need and leave us to eke out a primordial existence after the bottleneck of a mass resource collapse. “The humans would be living on the periphery, probably just dying of starvation and that sort of thing,” says Grace. It could be similar to how we treat other animals, like burning down the rainforests of Indonesia to make more palm oil for processed foods. “We don’t actually bother killing the orangutans” that live in Borneo, she says. But they are dying out anyway . She uses a verb I’ve never heard before: obsoleted . Leahy also sees the possibility of this annihilation as by-product: “I don’t think humanity will necessarily be the enemy of AI so much as that we will be collateral damage between billions of systems competing with each other in ways where humans just don’t have any input. We don’t have any economic, political, or military relevance.” Let’s say you watched a few videos on restarting an agrarian society fueled by the power of the sun; could a civilization reset thrive in the middle of nowhere or the alleys between the data centers? Unfortunately, Grace says, all the heat from the data centers would make the world “much more uninhabitable.” “If you start asking where are the physical limits of how much computing power you can generate on this planet, it’s heat dissipation,” says Soares. “If the AIs are trying to acquire all the resources to go take other planets, they prefer to run the planet very hot — at the maximum temperature at which you can still have a data center.” Which is 113 degrees Fahrenheit, nearly double the average surface temperature of the planet today. This post has been updated to correct a quote that was attributed to Nate Soares that was said by Connor Leahy. Tags: tech ai artificial intelligence Why the AI Apocalypse Won’t Play Out Like The Terminator Your product is saved! You’ll receive emails when your saved products go on sale. Manage preferences.