Who gets to say AI is safe? Closing the control gap
Key points
Key takeaways from Source Breaking Analysis episode 329 on the AI control gap, drawing on a conversation with Appian founder and CEO Matt Calkins (October 2026):
The control gap. There is a gap between what AI can do and what the evidence supports trusting it to do; the unresolved question is who gets to decide when the evidence is sufficient to declare a system safe, with independent oversight and enforceable responsibilities proposed as the answer.
White House accord is a start. The 29 September accord adds external assessments and oversight by an independent committee of each board, but it creates no public authority with compulsory access, binding conditions or penalties for non-compliance — so a paper signed by tech leaders is meaningless without a regime behind it.
A duty to prevent harm. Calkins compares AI to banks and power plants: industries capable of causing serious damage are required to behave responsibly upfront rather than pay civil penalties afterwards, and the cost of acting now is low because relatively little harm has been done so far.
China's pace depends on the US. Calkins argues China's AI progress largely rests on distilling American models and that Beijing has its own incentive to keep AI under control; even if China did catch up, the result would be comparable competition, not the end of history.
Alignment is the missing science. Alignment means AI doing what humans ask and obeying the law; the alternative is rogue AI that hacks or blackmails against instructions, and advancing the science of alignment is increasingly urgent as models grow more powerful.
Enterprise concerns are operational. Qualitate customer data flags stealth AI that bypasses established security controls and undocumented, so-called vibecoded agents that outlive the employees who created them, running unseen inside organisations.
A three-part oversight regime. theCUBE prescribes qualified independent experts defining public standards, expert assessors testing vendor evidence, and an accountable public authority able to require remediation or stop high-risk activity, with hurdles that scale with risk without pricing out smaller firms.
Oversight is a market opportunity. The operating challenge between models and consequential work — identifying agents, limiting access, recording actions and intervening — gives workflow, security and data-infrastructure vendors such as IBM and Oracle a commercial opening, since they are not waiting for the LLM labs to solve it.
Read more: theCUBE on YouTube
