Will AI really kill us all? The science behind the hype - Nature

Email Bluesky Facebook LinkedIn Reddit Whatsapp X Save article View saved research Credit: Daniel Megias/Getty Concerns that artificial intelligence might kill us all are blowing up right now following warnings from researchers and leaders at some of the world’s foremost AI companies. Speculation is rife over how legislators will act. Nature breaks down what the fears are, who is expressing them and what might happen next. How did the latest fears about AI begin? The latest hysteria about AI kicked off on 8 September, researcher Jacob Coxon told the Wall Street Journal that he was resigning from the AI firm Anthropic, based in San Francisco, California, because he feared that the systems the company is developing could spiral out of control and destroy humanity. In a subsequent post on the social-media platform X, his first ever, Coxon wrote: “The people building AI earnestly believe that it could kill us all by the end of the decade.” Hours later, Evan Hubinger, who leads Anthropic’s ‘alignment science’ — the effort to get model actions to correspond to human values — amplified the message by reposting it and add ing his own estimate that the risk of human extinction is “>10% within the next decade”. Coxon’s post racked up more than 100 million views in 24 hours. Dario Amodei, the firm’s chief executive, then posted an essay calling for a slowdown – but not a halt – in AI development. Rival AI leaders Sam Altman of OpenAI in San Francisco, and Elon Musk of xAI in Palo Alto, California, have since backed this suggestion. How exactly would AI kill us all? Coxon, Hubinger and Amodei have not given precise details of how AI might dispose of humanity. But there have been attempts to flesh out scenarios. In a speculative forecast called AI 2027, developed by the non-profit AI Futures project, an AI unleashes a biological weapon to kill humans off and thereby make more space for solar panels and robot factories. Researchers with such concerns tend to say that extinction is possible, if not inevitable, if two assumptions are made: first, that these systems will eventually completely outwit humans, and second, that the goals of such a system will not fully match those of humans. One famous example is that a ‘superintelligence’ that aims to manufacture as many paper clips as possible could end up making Earth uninhabitable, just to achieve its goal. Many researchers who worry about existential threats from AI “just assume that once we are at that point, the rest is details”, says Michael Vermeer, who researches science and technology policy at the RAND Corporation, headquartered in Santa Monica, California. Making such predictions usually “involves so many untestable claims that you just end up with a conversation that is really more like faith than something scientific or empirical,” says Vermeer. This makes basing any course of action on these conversations difficult, he adds. Instead, in 2025 1 , Vermeer and his colleagues looked at practical scenarios involving one of three existing technologies: nuclear weapons, biotechnology or deliberate modifications to the atmosphere. They found that complete extinction by nuclear weapons is not feasible, but the other two scenarios could not be ruled out. However, they also found that carrying these out would require models to have considerable ability to physically interact with the world, and that AI's murderous efforts would almost certainly take time and be detectable by humans, who might then stop their eradication. So how worried should we be? The problem is that AI models are largely probabilistic systems that reflect their web-scraped training data, says Heidy Khlaaf, chief AI scientist at the AI Now institute in New York City. They do not have human-like understanding and can achieve their goals through unpredictable shortcuts. Harmful behaviours have included attempting to blackmail people in test scenarios and hacking real-world companies — although the latter happened when safety guard rails were removed to test the systems' behaviour, and the models were given a task that incentivized them to seek unauthorized solutions. Many researchers worry instead that insufficient efforts by companies to limit and monitor the actions of their models will lead to damaging outcomes that are much more immediate and likely than is existential risk. These range from disinformation and psychosis to catastrophic events, such as enabling humans to deliberately create a bioweapon or even triggering a war. According to CNN , false information in an AI-generated report almost led the US military to board a Chinese ship earlier this year. Khlaaf, who has studied how AI is used in drafting regulatory documents for nuclear power plants, says that “AI's low reliability and accuracy rates in critical environments with life-or-death consequences” are of much greater concern to her than are threats of the technology wiping out humanity, which she calls “fear-mongering”. Why have the latest fears gone viral? Since 2023, Amodei and his peers have repeatedly made statements about their extinction concerns, but this time the conversation has entered the mainstream. A public backlash against the building of data centres and an effort from US legislators, including senator Bernie Sanders, to introduce a bill to ban ‘artificial superintelligence’ in the United States, might have added fuel to the latest fire. The past two months have also seen a flurry of concerning cybersecurity incidents , prompting almost 1,400 employees in the AI sector to sign an open letter calling for a slowdown. Credit: Saul Loeb/AFP via Getty Amodei and Hubinger’s involvement suggest that companies welcome the focus on AI safety. In his essay, Amodei says his fears have been heightened by the cybersecurity incidents and by the prospect of AI building its own successor, a process known as recursive self-improvement, which is being tried across the industry and which, he says, could reduce humans’ ability to understand and control the models. Coxon also referred to “systems that can improve themselves” in his resignation comments. Stricter regulation would allow Anthropic, which is reportedly close to making an initial public offering, to justify a slower pace of development without giving rivals an edge. Some have posited more reasons for the move. Anthropic and other firms face “massive product-liability exposure” if their models enable a “truly damaging cyberattack”, wrote David Sacks, co-chair of the US President’s Council of Advisors on Science and Technology, on X. Enjoying our latest content? Log in or create an account to continue Access the most recent journalism from Nature's award-winning team Explore the latest features & opinion covering groundbreaking research Access through your institution or Sign in or create an account Continue with Google Continue with ORCiD doi: https://doi.org/10.1038/d41586-026-02941-3 References Vermeer, M. J. D., Lathrop, D. & Moon, A. On the Extinction Risk from Artificial Intelligence (RAND Corp., 2025). Download references Reprints and permissions Related Articles AI is set to completely transform cybersecurity — here’s how researchers must prepare ‘It is incredible’: How AI is transforming mathematics We need a new ethics for a world of AI agents Who gets credit in the AI era? OpenAI maths bombshell sparks debate Why AI companies can’t be trusted to self-regulate Subjects Technology Information technology Policy Government Latest on: Technology Information technology Policy Why AI companies can’t be trusted to self-regulate World View 22 SEP 26 AI co-scientists are revolutionizing how research is done Technology Feature 21 SEP 26 Chinese companies doubled down on science after US tech restrictions News 18 SEP 26 Why AI companies can’t be trusted to self-regulate World View 22 SEP 26 Will wearable technologies transform clinical trials? Spotlight 16 SEP 26 Protecting education in the middle of war zones Correspondence 25 AUG 26 Biosecurity must protect human functional capacity Correspondence 22 SEP 26 A global safety net for climate-risk intelligence is urgently needed Correspondence 22 SEP 26 Why AI companies can’t be trusted to self-regulate World View 22 SEP 26 Jobs Associate or Senior Editor, Nature Communications Job title: Associate or Senior Editor, Nature Communications (Human Behaviour team) Locations: Shanghai, Berlin or Pune (hybrid) Closing date: 11th... Shanghai, Berlin or Pune (hybrid) Springer Nature Ltd Oncology, Cancer & Blood Diseases Institute - Cell & Immune Therapy – Physician-Scientist - JR214185 The Cancer & Blood Diseases Institute at Cincinnati Children’s invites applications for a faculty position in Cellular & Immune Therapy. Cincinnati, Ohio Cincinnati Children's Assistant Professor of Physiology (in the Institute of Human Nutrition) - Tenure Track The Institute of Human Nutrition (IHN) at Columbia University seeks candidates for Assistant Professor in the tenure track. New York City, New York (US) Columbia University Irving Medical Center / Vagelos College of Physicians and Surgeons Cancer and Blood Diseases Institute - Brain Tumor Research - Faculty - JR225336 The Cancer & Blood Diseases Institute, Cincinnati Children’s, invites applications for a faculty position in Brain Tumor research or CNS malignancies. Cincinnati, Ohio Cincinnati Children's Talent Recruitment Announcement at the College of Plant Science & Technology Gather Global Talents, Forge Great Achievements Wuhan, Hubei (CN) Huazhong Agricultural University (HZAU)