News

Anthropic’s AI advances trigger new safety fears on eve of IPO

Anthropic disclosed 'with little fanfare' in an update to its S-1 last week that it has activated part of its 2023 responsible scaling policy (RSP), based on its 'overall rate of progress'. The RSP mandates more oversight and security once Anthropic can 'compress two years of... AI progress into a single year', measured against the 2018-2024 pace. A METR report this month, written with access to unreleased models, estimates Anthropic 'made roughly two years of historical progress' in 12 months, with models improving about twice as fast as last year, largely because researchers use AI tools. The prospectus says 'each new version of a Claude model directly accelerates development of the next', calls this an advantage, and warns models could 'pose catastrophic or existential risks to humanity'. Critics have sharpened their tone: a coalition led by the Center for AI Safety, including founders of Hugging Face and xAI, said 'Anthropic is not a safe AI company' and accused it of using 'safety branding to court investors, employees, and the public'. Apollo Research CEO Marius Hobbhahn says 'safety is nowhere near ready'; the AI Futures Project expects a superhuman AI researcher 'at some point in 2027'; OpenAI board member Paul Christiano also voices concern. Anthropic's responsible scaling officer Sam McCandlish said on social media that many of METR's conclusions match his own, but that 'it's hard to say exactly how much' AI is speeding up research; the IPO quiet period limits further comment. Dario Amodei proposed a 'speed limit' on recursive self-improvement in September. Counterweight: Anthropic has not reached self-improving AI without human input; the policy was invoked on expectations about future models. The IPO priced at $83, above the guided range, raising up to $100bn at a valuation of about $1.5tn; shares trade from Thursday.

Why it matters

On the day Anthropic starts trading, its own filing confirms that its progress has crossed a self-defined safety trigger. That shifts the debate about AI-accelerated R&D from forecasts to a disclosed, auditable threshold. For anyone tracking frontier-lab valuations, this is now part of the risk case investors are buying into, and a template other labs and their backers will be measured against.

Executive takeaways

1) Anthropic activated part of its 2023 responsible scaling policy, disclosed in an S-1 update last week, because its 'overall rate of progress' met the RSP trigger of compressing two years of AI progress into one (versus the 2018-2024 pace). 2) METR, with access to unreleased models, estimates roughly two years of progress in 12 months, about 2x last year's rate, driven mainly by AI tools used by Anthropic's researchers. 3) The prospectus presents this self-acceleration as a competitive advantage while warning of 'catastrophic or existential risks'. 4) Critics (a Center for AI Safety-led coalition, Apollo Research, AI Futures Project, Paul Christiano) are escalating, with a superhuman AI researcher forecast for 2027. 5) Caveats: Anthropic has not reached autonomous self-improvement, the trigger is based on expected future models, and Anthropic's own safety officer says the speed-up is hard to quantify.

What Financial Times may be missing

The article doesn't say which parts of the RSP were activated, or what extra oversight and security measures now apply in practice, and it doesn't mention ASL levels. There is no reaction from regulators such as the EU AI Office, the UK AI Security Institute or California's SB 53 regime, and no view from IPO investors on whether this changes the risk disclosure they priced. METR's estimate depends on its methodology and on access granted by Anthropic, and the article gives no independent replication. Comment from Anthropic is limited by the quiet period, so the company's side rests on the filing and one social-media response. Comparable figures for OpenAI or Google DeepMind are not given, so it is unclear whether Anthropic is unique or simply the first to disclose.

Read the full article: Financial Times

More top stories