Human review recovered up to 8.49 points WebArena-Lite's evaluator missed; phoneshell's 52 failed cross-model runs kept nothing a reviewer could check

Two papers this week treat the evaluator's verdict as a first draft. One audits it by hand, the other refuses to collapse outcome and progress into one number. phoneshell grades from device state precisely to avoid trusting the model, and that only holds if its own checks can be checked.

A human audit of WebArena-Lite, posted on 1 October 2026, found the automatic evaluator had rejected 9 to 14 of GPT-5.5's successes in each of four settings, between 15.5% and 24.6% of the true successes. phoneshell's two public cross-model rounds on an iPhone contain 52 failed runs. In 33 the model said it had succeeded, and 16 of those sit on two tasks where the right app was open and only the final check failed. We cannot say how many were checker misses, because phoneshell kept nothing from the end of any run that a person could look at.

What the WebArena-Lite audit found

Chengguang Gan, Zimeng He, Yoshihiro Tsujii, Ken-ichiro Kobayashi, Hiroki Itoh and Kotaro Funakoshi (arXiv:2610.01491) ran all 165 WebArena-Lite tasks under six conditions: GPT-5.5 with a memory-and-analysis module (MASM), with and without task guides, at 15 and 25 steps, plus an untrained Qwen3.5-9B. They kept the evaluator's verdict, then had reviewers look at every failure through the actions, before and after screenshots, URL changes and final response. Review recovered 47 successes over the four GPT-5.5 settings. The causes were ordinary: a correct answer in a format the rule rejected, a state change the final rule did not look for, a correct result followed by an invalid stop.

The audit also tested the model's own account. Of 102 failures in the strongest setting, 18 claimed at least 80% completion in memory and 7 claimed 100%. Among the 40 that ran out of steps, the mean verified progress was 56.2% and none had verifiably finished. So review cut both ways: it found successes the evaluator missed, and it refused successes the agent asserted. Scroll oscillation was the largest single failure cause, 32 of 102.

MedImageOSWorld keeps outcome and progress apart

MedImageOSWorld (Long, Li, Xing, Yang and others; arXiv:2610.04800, 3 October 2026) runs agents on simulated CT, MR and ultrasound consoles and scores three things it never merges: success, workflow progress in quarters, and acquisition quality against hidden anatomy. Eleven open-weight agents reached 21.5 to 74.5 on progress and 3.0 to 25.0 on success. GPT-6 Sol and Claude Opus 5.5 reached about 34 success, with progress only 9 to 11 points above the best open model (85.5 and 83.4 against 74.5) but acquisition quality about 19 points higher (61.1 and 61.3 against 42.0). Its gates also check cause: on renal MR, Opus 5.5 restored full coverage in all 12 translation-defect cells and passed none, because it fixed the image through a setting other than the one at fault. In 360 CT recovery episodes from two models, no agent corrected the clipped range before closing the study.

phoneshell's 52 failures

The xv round (144 runs, 24 tasks, 6 models, 8 September 2026) and the xv2 round (349 runs, iPhone 14 Pro on iOS 26.3.1, 9 September) are on Hugging Face as blolabel/phoneshell-bench. Each task is graded by checks run against the phone after the agent stops: the foreground app, an element that must exist, or a regular expression over the text of the accessibility elements. We split the 52 failures by the model's final reply: 33 claimed success, 11 ran out of time or turns, 7 said they could not finish, 1 left no reply.

Two tasks hold half of the claims. On clock.timer.set, 11 of 12 runs failed across six models, every one with the Clock app in front and every reply describing a running five-minute timer, two quoting a countdown at 04:59. The check wants 0?[34]:\d\d|0?5:00 somewhere in the element text; the task file records that an earlier pattern had already failed a correct run, and the widened one still failed 11. On reminders.create in xv2, 5 of 6 models failed with Reminders in front and replies saying "collect parcel" had been added; one said the list count went from 7 to 8. These look like the WebArena-Lite pattern: a state the rule did not see. They could also be agents describing what they meant to do. The calculator replies show the second case is real: two runs reported 52 for (12 + 8) x 5.

If all 16 were checker misses, xv would read 86.8% instead of 82.6% and xv2 95.1% instead of 92.3%. That is an upper bound for two tasks, not a correction, and we cannot tighten it: the bench writes one result row per run, the runner never saves the final screenshot or accessibility tree, and the next task's setup changes the phone.

What it means if you grade agents from device state

• Device-state grading removes the model's self-report from the score. It does not remove the checker's errors, and those are now invisible unless the final state is saved.
• Save the final screenshot and element tree for every failed run, and every run where the reply claims success. On phoneshell that is one phone.snapshot() call before teardown, the call the checks already make, with the screenshot switched on.
• When one task fails across every model with the same claim, audit the check first. Eleven identical failures on one timer are more likely a pattern than six independent mistakes repeated twice.
• Report raw and corrected scores side by side, as both papers do, and keep progress separate from success. MedImageOSWorld's gap between 74.5 progress and 25.0 success is the information a single number throws away.

Check it yourself

• WebArena-Lite: Table 1 (raw, corrected, recovered), Table 6 (failure modes), Table 7 (claimed vs verified progress) of https://arxiv.org/abs/2610.01491.
• MedImageOSWorld: Section 5 (metrics) and 6.2 (results) of https://arxiv.org/abs/2610.04800; prompts and run configurations are announced for Hugging Face.
• phoneshell failures: download results/xv.jsonl and results/xv2.jsonl from huggingface.co/datasets/blolabel/phoneshell-bench and count passed == false and not skipped: 25 and 27. Filter task_id == "clock.timer.set" for the 11 failures and their check details. The success-claim split needs agent_said, which is on the rig only and not published, by design.
• Checks: environments/clock_timer_set.yaml and environments/reminders_create.yaml; how a regex check reads the screen: phoneshell/bench/verify.py, which joins e.text over the snapshot's elements.

What would prove this wrong

Our expectation is that at least half of the 16 same-task failures were checker misses. Test: rerun clock.timer.set and reminders.create 5 times each with two models, saving the final screenshot and element tree, and have a person grade every run from the saved state alone. If fewer than half of the runs the checks fail and the person passes appear, the checks were right and the agents were overclaiming, and we are wrong. We will publish the saved states and the grades by 31 December 2026, or say we did not run it.