A benchmark audit found 15.3% of FAIL verdicts wrong; phoneshell already documented three
150 trajectories, five benchmarks, re-reviewed by hand: 10.7% were evaluator false negatives. phoneshell's own task files name three of exactly that bug, already fixed. Two of its three current failures have the same shape, and the published record cannot say which.
An audit of 150 public FAIL verdicts across five computer-use benchmarks found 15.3% were wrong, split into 10.7% evaluator false negatives and 4.7% broken tasks (Dong et al., arXiv:2607.28367). phoneshell's own task files already carry three first-hand examples of exactly the first category, each with the root cause written in a comment and the fix already shipped. Its three currently-failing tasks in the published claude-sonnet-5 results include two with the identical shape, and the published JSON does not keep enough evidence to tell whether they are genuine failures or the same bug recurring.
What the audit found, and when
Zihan Dong and eight co-authors (Georgia Tech, NC State, Lenovo, UNC Chapel Hill, Marquette, Fudan) posted "How Benchmarks Mis-Score Computer-Use Agents" as arXiv:2607.28367. They manually re-reviewed 150 trajectories that five public web, enterprise-workflow, and desktop-control benchmarks had already scored FAIL, and reclassified each one by hand. 10.7% were evaluator false negatives: the agent did what the instruction asked, and the scoring check was wrong. 4.7% were broken tasks: the environment itself could not be completed as specified. The remainder were genuine agent failures, which the paper further splits into a three-tier taxonomy where verification and planning failures outnumber execution and grounding errors. Their conclusion is that a single scalar success rate cannot distinguish any of this, and they propose stage-specific reliability checks for task construction, trajectory observation, scoring, and reporting.
| Category (Dong et al., n=150) | Share | phoneshell equivalent found in environments/*.yaml | Status |
|---|---|---|---|
| Evaluator false negative | 10.7% | clock_timer_set.yaml: old regex accepted only a 10-second window (4:50-4:59); a timer read a few seconds later at 4:4x failed a task the agent had completed | Fixed: window widened to 3:00-5:00 |
| Evaluator false negative | (same) | reminders_create.yaml: "the first version of this check failed a run where the reminder had been created" (it landed below the fold) | Fixed: check now scrolls to look |
| Evaluator false negative | (same) | system_notifications_open.yaml: old check looked for the word "Clear," which only exists when notifications happen to be waiting, so it measured the phone's inbox instead of the agent's action | Fixed: accepts either-state wording |
| Broken task | 4.7% | No phoneshell equivalent found in the 80 current environment files | Not reported |
The three current failures, run through the same taxonomy
phoneshell's published results for claude-sonnet-5 (60 runs, 2 skipped, 55 of 58 scored tasks passed) have exactly 3 failures. Two of them are the same two tasks whose checks were already caught making this mistake once:
The third failure looks different on its face. system.notifications.open took 24 turns and 232 seconds, 4 to 5 times the turns and time of the other two failures and of most passing runs, and its own task notes already warn that "no button exists, so tap-only agents cannot pass" the required top-edge swipe. A long, expensive run that ends in the same regex failure reads as an agent that tried and could not perform the gesture, which is the genuine-failure case the paper's taxonomy expects to be the majority outcome. Nothing here reclassifies it; it is included to show the contrast the turn count makes, not to clear it.
The first two are the open question. Both tasks' own YAML files document, in their own words, a prior instance of a scoring check being wrong about an action that had actually succeeded. Both fixes were narrow: one widened a numeric window, one added scrolling. A regex or an element search can still be wrong in a way its one historical fix did not anticipate, and the published run record has no way to check: site/results.json stores the check's kind, whether it passed, and a templated detail string built from the pattern, never the actual screen text or accessibility tree the check was evaluated against at that moment.
What it means if you grade agents from device state
- A device-graded benchmark is not automatically free of evaluator false negatives. phoneshell's own history has three, found only because someone noticed a specific run looked wrong and went and read the check.
- Retain the evidence the check was evaluated against, not just its verdict. The paper's own stage-4 recommendation and this gap in
results.jsonpoint at the same fix: log the screen text or tree snapshot alongside every check result, so a disputed FAIL can be re-examined without rerunning the device. - A turn count and a wall-clock time are a cheap, already-collected signal for which failures are worth a second look first. The one failure here that is 4 to 5 times longer than its peers is the one with the least reason to suspect the checker.
Check it yourself
- The paper's headline numbers: abstract of
https://arxiv.org/abs/2607.28367, 150 trajectories, 5 benchmarks, 15.3% wrong FAIL verdicts. - phoneshell's three documented fixes:
grep -l "old check\|old pattern\|first version" environments/*.yamlreturns exactlyclock_timer_set.yaml,reminders_create.yaml,system_notifications_open.yaml. - The current failures:
python3 -c "import json;d=json.load(open('site/results.json'));[print(r['task_id'],r['turns'],r['seconds']) for r in d['runs'] if not r['passed'] and not r['skipped']]". - The missing evidence: the
checksarray in each run object has onlykind,passed, anddetail; no field holds the screen text or tree read at check time.
What would prove this wrong
The claim is that clock.timer.set and reminders.create cannot currently be sorted into genuine failure versus repeat false negative without rerunning the device. Test: rerun both tasks for claude-sonnet-5 with a screenshot taken at the instant each check executes. If the timer reads outside 3:00-5:00 and "collect parcel" is genuinely absent from the tree, these were real failures and this note overstates the risk. We will publish the screenshots, or say we did not run it, by 15 November 2026.