News

Ramez Naam: real-world data from OpenAI and Google shows the most-cited AI progress charts overstate capability by 30-100x

What's the argument? Author and investor Ramez NaamDealroom has a profile for this one. Try Dealroom → argues that the most influential charts of AI progress, METR's task-length chart, extrapolations of it and the AI 2027 forecast, badly overstate what AI can do in real work. He bases this on new data OpenAI and Google DeepMind have published on how AI models perform on research tasks inside their own companies.

By the numbers: On internal tasks, AI models succeed 80% of the time on tasks of around nine minutes at Google and an adjusted 16 minutes at OpenAI, Naam calculates. At that success rate, estimates derived from METR's benchmark are 33 times OpenAI's internal task length and 41 times Google's, and the original AI 2027 scenario overshoots by more than 100 times.

The gap is widening: Between June and October, the gap between forecasts and OpenAI's real-world results roughly tripled, he finds. Real-world task length does now look to be growing exponentially, doubling about every 3.9 months at 50% success and every 5.6 months at 80%, but slower and less reliably than the forecasts assume.

Why it matters: Naam's conclusion is that AI is "much more capable than it is reliable". Fully autonomous self-improvement would need very high reliability on tasks lasting days, which these numbers suggest is still many years away. He calls for a transparency "race to the top", praising OpenAI for publishing its data and criticising Anthropic for having METR redact evidence from a report on its internal models. He stresses that he respects the forecasters, who built their models before real-world data existed.

Sources: Ramez Naam: The Most Important Charts in AI are Wrong
Ramez Naam on X

Image credit: Ramez Naam

More top stories