CoArena: Crowdsourced benchmark for computer-use agents
What’s the deal? CoArenaDealroom has a profile for this one. Try Dealroom →, formerly known as Coasty, is developing a crowdsourced benchmark for computer-use AI agents. Its product video contrasts polished, staged demonstrations with live trials in which two agents attempt the same real task without a script, edits or second takes. The company describes the benchmark as a way to evaluate whether agents can complete practical computer workflows reliably rather than merely produce convincing demos.
How it works: CoArena runs agents on real-world tasks and has people judge the results. The video cites 10,000 agent runs and 17,000 human verdicts, and presents the service as an arena where frontier AI models are ranked by work assessed by real users. Its website describes live tasks, blind human judgment and published numbers accompanied by the rule that generated them. This model is intended to make evaluation criteria visible and comparable, while capturing failures that may not appear in static benchmarks or carefully prepared product demonstrations.
Why it matters: Computer-use agents are expected to handle multi-step actions across websites and software, but their performance can vary substantially by task, interface and recovery requirement. A benchmark based on independent runs and human outcomes can provide a more direct view of that reliability. CoArena’s example video uses an agent called Aura attempting activities such as booking flights, filing taxes and planning travel, then frames the resulting trial as evidence about how models perform on actual work. The platform’s current positioning is therefore centred on practical agent evaluation, crowdsourced testing and transparent reporting for developers and teams deciding which computer-use systems are ready for broader deployment.
Read more: CoArena product video · CoArena