Computer Agent Arena's 2,201 human votes reverse agent rankings that static benchmarks report; phoneshell's own static leaderboard failed a different way, it could not separate three models at all

A leaderboard can mislead in two different directions. It can confidently rank agents in an order a human user would reverse, which is what Computer Agent Arena measured. Or it can confidently fail to rank them at all, which is what a McNemar analysis found on phoneshell's own six-model board. Neither failure looks like the other from the outside.

Computer Agent Arena ("Toward Human-Centric Evaluation and Analysis of Computer-Use Agents," ICLR 2026, presented 23 to 25 April 2026 in Rio de Janeiro) is an open-source platform, built by a 19-author team including Bowen Wang, Tao Yu, Victor Zhong, Yi Zhang, Diyi Yang and Ion Stoica, that runs two anonymized agents head-to-head in matched, controlled cloud environments and collects human preference votes rather than relying on a fixed task-completion script. Across 2,201 high-quality votes over 12 agents, the paper reports "striking ranking reversals relative to static benchmarks." Overall correctness mainly drives human preference, but the authors find that agent-human interaction and self-correction also raise a user's preference even when overall task completion looks similar, and their error analysis names failure categories, long-horizon memory failures and fine-grained action failures among them, that the authors say static benchmarks do not evaluate.

What changed, and when

Computer Agent Arena's critique of static benchmarks is about validity: a benchmark can produce a confident, statistically clean ranking that still does not match what a human user, watching the same agents work, would actually prefer. Its fix is to go get that preference directly, at scale, rather than assume task completion alone captures it. The platform and its 2,201-vote dataset are reported as open-sourced alongside the paper.

phoneshell's own leaderboard failure, documented in FINDINGS.md section 28, is about statistical power, a different axis entirely. Six models ran the identical 24-task public suite, a paired design, and a McNemar analysis of the pairwise disagreements found that three of the six, Claude Opus 4.8, GPT-5.1 and Qwen3-Max, were exactly symmetric: each won the same two tasks the other two lost. The published board had shown an 83.3% tie between them as if it were a ranking; the correct read is that the instrument could not tell them apart, and no size of sweep would change that, because there was nothing in the disagreement pattern left to scale. McNemar needs six one-directional disagreements to reach p < 0.05 regardless of sample size, and these three pairs never had more than two.

Reversed versus absent

What it means for someone grading computer-use or phone agents

Both results point the same reader toward the same discipline, approached from opposite directions. Computer Agent Arena's lesson is that passing a validity check, agreeing with human preference, is a separate question from passing a statistical one, separating cleanly on a fixed task set; a ranking can be statistically tidy and still wrong about what matters. phoneshell's section 28 lesson is the mirror case: a ranking can look tidy, sorted percentages descending, and still carry no statistical content at all for part of the field, which is a problem a human-preference study would not even get the chance to contradict, because there was no real ranking there to compare against in the first place.

Neither project has run the other's check on its own data. Computer Agent Arena does not report, at least in what is available here, whether its reversed rankings would also fail a power analysis like phoneshell's McNemar test; and phoneshell has never collected a single human preference vote on any of its 80 tasks, so whether its device-state grades would agree with what a user watching the same run would prefer is simply unmeasured. A model that passes phoneshell's accessibility-tree check might still take an interaction path a human found clumsy, slow, or oddly phrased, properties phoneshell's binary pass or fail was never built to capture and Computer Agent Arena's whole platform exists to collect.

• A statistically significant ranking and a human-preferred ranking are two different claims. Computer Agent Arena shows they can diverge; phoneshell's section 28 shows a third failure mode sits underneath both, a ranking that was never statistically there to diverge from anything.
• Error categories that static, completion-only benchmarks miss, Computer Agent Arena names long-horizon memory failures and fine-grained action failures, are exactly the kind of thing a device-state check like phoneshell's can sometimes see (a wrong value on screen) and sometimes cannot (an agent that got the right answer by a clumsy, low-trust route).
• Before trusting any sorted leaderboard, phone or desktop, ask both questions separately: does the order survive a paired significance test (phoneshell's McNemar fix), and does the order match what a human doing the same task would actually prefer (Computer Agent Arena's whole premise). A board can fail either one without failing the other.

Check it yourself

# Computer Agent Arena's abstract and reported vote count
curl -s https://proceedings.iclr.cc/paper_files/paper/2026/hash/5f1d328711b162e1cf9896c7d6aaecdd-Abstract-Conference.html

git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "## 28" -A40 FINDINGS.md | head -60   # the three-way tie and the McNemar arithmetic
cat phoneshell/bench/stats.py                  # Wilson intervals, McNemar exact test, power calc, in one place
python3 scripts/run_sweep.py --help            # the resumable, round-robin sweep the fix produced

What would prove this wrong

• We claim phoneshell has never collected a human preference vote on any run. That is wrong the moment such a study exists and reports whether its ranking agrees or disagrees with the device-state grades on the same tasks. We predict that on phoneshell's render-stratum tasks specifically, where the pass or fail condition is an unambiguous, narrow fact, human preference and device-state grading would agree closely, because there is little room for a "clumsy but correct" path to diverge from a "clean and correct" one on a single-fact check. On phoneshell's more open-ended tasks, we would expect more disagreement. Untested as of this writing, falsifiable by whoever runs it, through 31 December 2027.
• If Computer Agent Arena's full paper, read beyond its abstract, reports a power analysis of its own 2,201-vote comparisons, that would answer whether its reversals also survive the same significance standard phoneshell now applies to its own board, and we would fold that number in here.