OpenAI publishes its internal pretraining curve and proposes industry-wide limits based on perplexity
What's new? OpenAI chief scientist Jakub Pachocki has published the base-model log-perplexity of OpenAI's models from GPT-3.5 onwards, the internal metric the lab has used for years to track pretraining progress. In an essay titled "Perplexity and Intelligence", he proposes that frontier labs adopt perplexity as a shared yardstick and agree on common limits they would not cross until the ability to understand and monitor AI behaviour reaches concrete milestones.
What the numbers show: Perplexity measures how well a model predicts the next token; lower is better. On OpenAI's main held-out dataset (drawn from its internal codebase), GPT-3.5 scored about 0.99 and GPT-4 about 0.91. Below 0.8, Pachocki writes, OpenAI saw transformative capabilities emerge in cybersecurity and mathematics, and once a model reached about 0.76 it started producing "groundbreaking" results on open maths problems. The chart shows a steep drop through 2026, with an unreleased internal model plotted lower still. OpenAI attributes the 2026 acceleration mostly to long-term research and compute investment, and only in smaller part to AI automating its own development.
The proposal: Pachocki expects further transformative, as yet unknown, capabilities in the 0.68 to 0.70 window, alongside weaker alignment auditing and models more aware of when they are being evaluated. He argues that AI could trigger an "avalanche" of pretraining advances as soon as 2027, faster than alignment and monitoring are likely to keep up. He wants labs to set shared evaluation and transparency norms, and an industry standard that fixes a first limit on development, measured by a metric tracked across labs, plus what must be achieved (such as a successor to chain-of-thought monitoring) before going beyond it.
The caveats: He acknowledges that perplexity is hard to compare across labs that use different datasets, does not map directly to useful tasks, ignores gains outside pretraining such as chain-of-thought reasoning, and has an unknown floor. Publishing the figures also gives competitors a clearer view of where OpenAI stands, which he says is outweighed by the urgency of shared safety measures.
Why it matters: It gives an unusually detailed public view of a frontier lab's pretraining progress, and it puts a concrete, measurable proposal for pacing AI development on the table at a time when calls for pauses and limits on frontier training are growing.
Sources: OpenAI: Perplexity and Intelligence
Jakub Pachocki on X
Image credit: OpenAI