Anthropic Warns AI May Kill Us All in $2 Trillion IPO Pitch - New York Magazine

Photo: Jason Henry/Bloomberg/Getty Images Earlier this month, Anthropic CEO Dario Amodei published a 4,000-word essay warning about the dangers of artificial intelligence, estimating that a bot “swarm could be capable of taking over the entire internet” in 6 to 12 months and that “the scale of damage would continue to increase from there if AI becomes more powerful.” Now, he’s back with another, graver warning about the havoc AI could wreak on humanity — while asking would-be investors to fork over billions of dollars and value his company at some $2 trillion. Yes, the company that is largely responsible for foisting AI on the public. Reuters reports that Anthropic’s IPO prospectus tells potential investors that artificial intelligence may pose “catastrophic or existential risks to humanity.” Among other pants-pissingly terrifying scenarios, the IPO reveals that the technology, which is getting smarter every day, could eventually “resist shutdown,” “conceal or manipulate information,” and engage in conduct “resembling blackmail.” The company also warns that its “development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm.” In other words, simply by continuing to operate, Anthropic, by its own admission, is putting humanity at further risk. The IPO also notes that as models get more sophisticated, they become more difficult to effectively monitor, because they understand that they’re being watched and become stealthier. “Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety,” the company states. According to Reuters, Anthropic “devoted roughly 80 pages of the … prospectus to laying out risk factors, nearly twice the 48 pages it used to describe its business.” Now, you might be reading this and thinking, Okay, all of that sounds pretty bad . Is this Anthropic’s way of announcing it’s bowing out of the AI arms race? That, knowing just how bleak things can get — and according to one of its former employees, AI potentially taking out humans by 2030 — the company doesn’t want to be part of the hellscape that’s coming? The answer: no! Not only is Anthropic not getting out of the business of AI, it wants to double down on the technology, and all it needs are billions of dollars from Wall Street. The prospectus is, of course, about convincing would-be shareholders to part with large sums of money to keep Anthropic going (and enrich everyone who got in early). In related news, it turns out the business of AI has not been entirely lucrative to date. In addition to detailing the various ways the technology could turn on humans, the prospectus also reveals that Anthropic lost $42 billion last year . It also told potential investors that almost one-quarter of the revenue it did bring in came from just two customers and, per Reuters, that “many of its largest clients were not locked into long-term contracts and could cut or stop spending.” Which doesn’t seem like, say, the best business model. Anyway, yes, Anthropic would like people to give it lots of money when it goes public — likely in early November — despite the fact that its CEO has become the public face of warning of a possible AI apocalypse. So much so that he was parodied last weekend on SNL : Is Amodei wrong about wanting to put safeguards around AI and slow down the development of the technology? Probably not, given the alternative. That’s an argument he’s likely pitched over the last several days, particularly during a dinner with Donald Trump on Sunday (and a lunch back at the White House today with fellow tech CEOs). Unfortunately, while few details have emerged from the private meal, there’s a high chance it was less than productive, assuming the president is still operating under the assumption that “the only control or ‘guardrail’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!” Tags: wall street anthropic dario amodei Anthropic Warns AI May Kill Us All in $2 Trillion IPO Pitch Your product is saved! You’ll receive emails when your saved products go on sale. Manage preferences.