Sen. Warren asks Anthropic CEO to address existential risks before IPO - Spectrum News

Following Anthropic’s filing for what is expected to be a blockbuster initial public offering of shares in one of the country’s top artificial intelligence companies, Sen. Elizabeth Warren, D-Mass., is asking its CEO for answers about potential existential risks to humanity. On Monday, the company said its advanced AI systems could pose a catastrophic threat without proper oversight, safeguards and security measures. “I believe the risks presented by this technology demand greater scrutiny before they become intertwined with the public capital markets, which could further escalate both public safety and systemic financial risks,” Warren wrote in an 11-page letter to Anthropic’s Dario Amodei that cited multiple excerpts from the preliminary registration statement Anthropic filed with the Securities and Exchange Commission this week, 90 pages of which were devoted to risk factors. Warren, the top Democrat on the Senate Banking, Housing and Urban Affairs Committee, cited Anthropic’s acknowledgment that its own employees have raised concerns about its AI systems being misused, replicated, weaponized or allowed to operate with high degrees of autonomy. She also highlighted Anthropic’s statement that its AI models could exhibit self-preserving behaviors that resist attempts to shut them down, while some of its models have already demonstrated capabilities that could meaningfully assist malicious actors who are trying to develop biological or chemical weapons. “I am deeply troubled by the risks raised by Anthropic itself and question how it can act in a manner consistent with its stated public benefit and related fiduciary duties under Delaware law without mitigating these risks to humanity,” Warren wrote. The San Francisco-based AI company behind the Claude chatbot, Anthropic said in its filing this week that it could “help humankind to chart a safe course through its technological evolution, open new frontiers of discovery, and bring forth an age of miracles.” But Warren said the company’s sheer size and dramatic changes in share price could roil financial markets once its shares are publicly available and make their way into retirement accounts. “This massive influx of public capital is likely to catapult the development and deployment of your technology during a time of intense public debate and concern about the risks of your frontier intelligence models,” she wrote in a letter that also asked the company to respond to 10 multipart questions by Tuesday. Anthropic’s IPO could take place around the week of Nov. 16, according to Yahoo Finance, in accordance with SEC rules that require a 15-day waiting period after a public registration filing. Financial analysts expect the Anthropic IPO to be the largest in history, raising up to $100 billion. The previous record was set earlier this year with a SpaceX IPO that raised $75 billion. Analysts value Anthropic at $1.5 trillion to $2 trillion, making it one of the most valuable companies globally. The Anthropic IPO will come at a time of public skepticism toward the technology. An ABC News/Washington Post/Ipsos poll this month found that 57% of Americans think AI is bad for society compared with 42% who think it is good. A poll from Politico last month found that 61% of Americans think there’s a risk that AI will destroy humanity, while 17% said it is almost certain to happen. Warren asked Amodei to outline the steps Anthropic is taking to ensure its technology will not cause an existential risk to humanity and to address concerns that AI will enable the development and use of biological weapons. She also asked how Anthropic might report incidents of AI models autonomously conducting cyberattacks and if incidents have already occurred that the company hasn’t publicly disclosed. In July, the company said its AI models hacked into three organizations during testing, prompting it to launch a large-scale cybersecurity review that specifically looked for evidence that its AI models were able to access the internet from within testing environments that should have been sealed off, the company said on its website. Anthropic did not respond to Spectrum News’ request for comment about Warren’s letter before this story was published.