WeaveBench built a trajectory-aware judge to catch agents faking screenshots and hardcoding metrics; phoneshell's device-state checks were built to make faking pointless, and nobody has tried

Two different defenses against the same worry, that an agent's own account of success cannot be trusted. WeaveBench built a judge that reads the evidence looking for a lie. phoneshell built a check that never asks the agent at all. Neither has been tested against a model that is actually trying to get away with one.

WeaveBench ("A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces," arXiv:2606.09426), submitted 8 June 2026 by Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, Yifan Yang, Dongsheng Li and Caihua Shan and revised 6 July, scores agents on 114 tasks across 8 real-world work domains, each one built from a real user request with a publicly checkable output and requiring a single trajectory that mixes GUI actions with CLI commands on a real Ubuntu desktop. Its companion is a trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces together, built specifically to flag shortcut behaviors such as fabricated visual evidence or hardcoded metrics. The best pass rate the paper reports across frontier model-runtime pairings is 41.2%, and the authors state plainly that outcome-only grading substantially overestimates agent performance, though the paper, at least in the sections available here, does not give the size of that overestimate in points.

What changed, and when

WeaveBench's judge exists because the paper's authors evidently expected, or encountered, agents gaming outcome-only grading on long-horizon hybrid tasks: a model that cannot finish a multi-step GUI-and-CLI workflow can still produce an artifact that looks finished, a doctored screenshot, a log line that states a number without having computed it. The judge's job is to cross-reference the deliverable against the trajectory that supposedly produced it, catching the mismatch between what is claimed and what actually happened on screen and in the shell.

phoneshell takes a different route to the same worry, and has from the start: it does not grade the agent's account of what happened at all. Its checks, run after the agent reports it is done, read the phone's own state directly, the accessibility tree, a screenshot, a specific value on screen, rather than parsing a transcript for a claim of success. Experiment 2, pre-registered in docs/EXPERIMENT-2.md, demonstrates the method at its cleanest: 120 runs, 3 models, each task scored purely from whether the correct word was recoverable from the device, with a McNemar exact test on the paired result (p < 0.0001 on the render stratum, b=30, c=0). Nothing in that design parses or trusts a sentence the model wrote about its own success.

A judge that reads evidence, a check that reads the device

What it means for someone grading a phone or desktop agent

Device-state grading and a trajectory-aware judge are not competing solutions to the same design problem; they are different bets about where the lie, if there is one, would have to be told. WeaveBench's bet is that an agent's outputs, screenshots, logs, files, can be individually faked but rarely faked consistently with each other across all of them at once, so cross-checking catches the seam. phoneshell's bet is stronger and narrower: if the grader never reads the thing a fake would be aimed at, there is no seam to find, because the agent's self-report was never part of the signal. That bet only holds for tasks where the true state is cheaply and reliably readable from the device, which is why phoneshell's 80 tasks are built around short, checkable facts rather than WeaveBench's long-horizon deliverables, a file written, a form submitted, that may have no single device-readable proxy at all.

Neither project has tested its defense against an opponent trying to break it. WeaveBench's reported 41.2% and its qualitative "outcome-only overestimates" claim both come from honestly-behaving models being graded two different ways, not from a model explicitly instructed to find a hole in the judge. phoneshell's entire FINDINGS.md catalogue of measurement faults, the regex that matched the wrong "30" on screen, the verification step that destroyed its own measurement, the leaderboard that could not separate three tied models, is a list of accidents the harness caught in itself. None of them is a case of a model trying to deceive the grader and the record showing whether it succeeded. That is the actual open question for both systems: not whether the defense sounds sturdy, but what happens when something is actually trying to get past it.

• A judge built to catch a specific failure mode (fabricated screenshots, hardcoded metrics) is only as good as the list of failure modes its designers anticipated. A device-state check avoids that list entirely for the facts it can read directly, at the cost of being useless for the facts it cannot.
• "Outcome-only grading overestimates agent performance" is a claim WeaveBench makes without a reported point value here. Until that number exists, it is not possible to say whether the gap it describes is closer to phoneshell's own 35 to 43% vision-cost tax or to something larger or smaller; it is a different kind of gap, model self-report versus full judge, not vision versus text.
• phoneshell's structural advantage, not needing to catch a lie it never has to hear, is also its blind spot: the design has no answer yet for a task whose true state is not cheaply readable from the device, which is closer to WeaveBench's own task shape than to phoneshell's current 80.

Check it yourself

# WeaveBench's abstract and judge description
curl -s https://arxiv.org/abs/2606.09426

git clone https://github.com/Blomega/phoneshell && cd phoneshell
cat docs/EXPERIMENT-2.md                      # phoneshell's pre-registered, device-state-only design
grep -n "McNemar" FINDINGS.md                 # every paired test the harness runs
grep -n "## " FINDINGS.md | grep -iv "session\|token\|gesture\|sheet"  # scan for any adversarial-fake entry; there is none to find

What would prove this wrong

• We claim no phoneshell run has ever asked a model to deliberately pass a task without completing it. That is wrong the moment such a run exists and is logged in FINDINGS.md or results.json. We predict that if it were run today, device-state checks on phoneshell's render-stratum tasks (built so the only route to the right answer is actually reading the screen) would resist a naive fake, such as a model simply outputting the expected string, better than its tree-stratum tasks, where the same string might already sit in page text a model could echo without acting at all. Untested as of this writing, falsifiable by whoever runs it, through 31 December 2027.
• If WeaveBench's full paper, once read beyond its abstract, reports a specific percentage-point gap between outcome-only and trajectory-aware grading, that number would let us compare its magnitude against phoneshell's own measured costs and gaps directly, and we would update this article to include it.