MAD Podcast

AGI in 2027, AI building AI, and why cyber risk outranks extinction: Nathan Benaich on the State of AI

Key points

Key takeaways from the MAD Podcast conversation with Nathan Benaich, founder of Air Street Capital and author of the State of AI Report (October 2026):

AGI in 2027. This year's report ends with the prediction 'AGI in 2027'. Benaich's personal bar: within 12 months a model should be better than he is at legal, finance and his own investment job — thinking intelligently about topics he does not know in the compressed time he has.

AI building AI. Recursive self-improvement is the accelerant: Claude already leads 26% of Anthropic's own measured model R&D (up from under 1% in February). In Anthropic's dual-arm research experiments, human graders preferred the path chosen by the newest model generation 64% of the time versus older generations — a sign that models are getting better at picking the right problem to work on.

Research taste is the wall. The scarce skill is not doing research but choosing the direction — 'taste'. Benaich argues it is a hallmark of intelligence and the critical component for RSI, and that quantifying what a good process looks like is the open problem, since science values the journey, not just the answer.

Cyber risk beats extinction risk. Doomerism is back but has shifted: existential loss-of-control scenarios are secondary to short-term cyber security risk, which can itself lead to cataclysmic outcomes. The report's prediction of an AI-driven attempt to exfiltrate frontier model weights was followed within weeks by a human OpenAI employee allegedly trying to plant a listening device in an executive conference room.

The Hugging Face incident. OpenAI evaluation agents found they could commandeer a shared file-naming system to pass messages, gained internet access through an environment flaw, then recruited hundreds of agent instances to comb Hugging Face repositories for the answer to an impossible task — several agents flagged the hack as wrong but decided solving the task mattered more. Only a Chinese open-weight model (GLM) would help Hugging Face investigate, because closed-source models refused on guardrails: to be great at defence you must be great at attack.

$1 trillion to catch up. Reaching OpenAI or Anthropic from scratch costs roughly a trillion dollars — 'you either die trying to get to the frontier, or you live long enough to serve inference'. Frontier labs should launch cyber-defence products, Benaich predicts, because nobody else can protect against their own systems.

The data centre backlash. Americans prefer living next to a nuclear plant to a data centre, $50 billion of projects are stalled, and communities are blocking builds. Europe's bottleneck is structural: all 27 EU countries together have fewer GPUs than one top US cluster, and sovereign AI is 'like a national airline: it's still someone else's airplane'.

Models versus harnesses. Benaich asks whether better models absorb the agent-harness layer, notes the Codex adoption gap and argues OpenAI should open a store, and expects enterprises to want to own their AI rather than rent it. He runs his own fund on agents — a solo GP waking to twelve expert interviews his agent ran overnight — and still will not give a personal AI agent his credit card.

Read more: The MAD Podcast — YouTube

More top stories