16 of 27 failures on a physical iPhone were reported as successes by the model that failed them

Grading an agent by what it says it did does not add noise. It adds a bias with one sign. On two paired sweeps, the self-reported score ran 4.5 and 11.1 points above the device's verdict, and none of the 441 passing summaries claimed a failure.

On 9 September 2026 six frontier models ran 349 tasks on a physical iPhone 14 Pro running iOS 26.3.1, and the phone's own state graded 322 of them as passed. Of the 27 failures, 16 ended with the model's summary saying the job was done. Graded by those summaries, the sweep would have reported 96.8% instead of 92.3%. Across that sweep and the 24-task run the day before, not one of the 441 runs the device passed had a summary admitting failure.

What was measured, and what other benchmarks grade with

The sweep is the one behind our 10 September note: Kimi K3, GPT-5.1, Claude Opus 4.8, Qwen3-Max, Gemini 3.1 Pro and DeepSeek v3.2 on 58 first-party iOS tasks, run between 05:06 and 12:28 UTC on 9 September 2026. Each task ends with a final phone_done summary written by the model, then checks that read the phone directly: foreground bundle, an element found after scrolling, a switch's value. The model never sees the checks. For this note we set every failed run's summary beside its check results and labelled each by hand.

The two largest 2026 mobile benchmarks grade differently. iOSWorld (arXiv:2606.09764, 8 June 2026) scores each trajectory with a GPT-5.4 Mini judge against a per-task rubric and reports kappa 0.77 and 89% task-level accuracy against human labels on 128 Opus trajectories. AndroidDaily (arXiv:2605.27761, 26 May 2026) grades 350 tasks across 94 closed-source apps with GRADE, a guideline-grounded reviewer at 87.37% agreement with human evaluators. OSWorld, per Epoch AI's 30 October 2025 review, checks "whether it has achieved a specified target state", which is the design used here.

Every failure, sorted by what the model said

The sixteen claims on the 9 September sweep, by task: reminders.create 5, clock.timer.set 5, system.notifications.open 3, and one each on calculator.chain, notes.type_long and resilience.scrolled_list. By model: GPT-5.1 4, Claude Opus 4.8 3, Qwen3-Max 3, DeepSeek v3.2 3, Gemini 3.1 Pro 2, Kimi K3 1.

How a summary gets ahead of the phone

Three of the claims show the shape clearly. GPT-5.1 reported the wrong number itself, 52 for an expression that equals 100, and still called the task done. Claude Opus 4.8 cited real evidence, a list count moving from 7 to 8, for a reminder whose title the device could not find after scrolling. FINDINGS.md section 16 records the same count rising for a reminder that was saved empty: keystrokes sent with no focused field are discarded while the keyboard still animates in. And DeepSeek v3.2 described the top level of Settings as the General page. In each case the summary is internally consistent and wrong about the world, which is the case a transcript-reading judge is least equipped to catch.

One cluster is contested, and we are not counting it as settled. Five of six models reported a running five-minute timer; Claude Opus 4.8 quoted "03:15 with a Pause button". The check reads the accessibility tree for a countdown like 3:15, and it did not match. We could not verify from these rows whether the timer was running with its value exposed differently, or not running. Leave clock.timer.set out and the count is 11 claimed of 22 failures on 9 September and 9 of 18 on 8 September, still half.

A correction to our own sweep: three failures were the rig's

Reading every failure also found three that should not have been scored at all. Claude Opus 4.8 on resilience.wrong_app, Gemini 3.1 Pro on settings.bluetooth.reach and Kimi K3 on weather.open each ended with the model saying WebDriverAgent had stopped answering, and each check detail reads "could not read the phone". Our rule is that an environment fault is never a capability result. The runner enforced that when the rig was down before a task started, but not when it died during one. Removing the three cells changes no conclusion in the 10 September note: the widest pair is still Kimi K3 against Qwen3-Max at 4 to 0, p = 0.125, on 56 tasks instead of 58.

What this changes if you evaluate agents

Treat a completion claim as an input to audit, never as a grade. On these runs it was wrong about the world in 16 of 27 failures and never wrong in the pessimistic direction, so any score built on it is an upper bound. The inflation is largest exactly where a benchmark still has headroom: 11.1 points on the harder 24-task run, 4.5 on the easier 58-task one.

Put state into the judge's evidence. Judgement, not measurement: a trajectory judge like iOSWorld's, at 89% agreement with humans, would see the Reminders count go from 7 to 8 and read it as confirmation. Only a lookup for the reminder's text, after scrolling, catches the empty reminder. An 11% judge error rate is also wider than the 6.9-point spread separating all six models on our sweep, 89.7% to 96.6%.

Make the unreachable state a first-class outcome. A check that returns "could not read the phone" is a measurement of the rig. It belongs with rate limits and locked phones as not scored, whether the rig failed before the task or during it.

Read the summaries anyway. They are poor grades and good diagnostics: all three rig outages and the five contested timer claims were found by reading what the agent said.

Check it yourself

The public export has every run's pass flag and check details, but not the model summaries, which were stripped from these two files before publishing. The 27 failures and their check text reproduce from the dataset:

curl -sLO https://huggingface.co/datasets/blolabel/phoneshell-bench/resolve/main/results/xv2.jsonl
python3 - <<'EOF'
import json
rows = [json.loads(l) for l in open("xv2.jsonl") if l.strip()]
print(len(rows), "runs,", sum(r["passed"] for r in rows), "passed")      # 349 runs, 322 passed
for r in rows:
    if not r["passed"]:
        print(r["model"], r["task_id"], "|", r["error"], "|",
              [(c["kind"], c["passed"], c["detail"][:60]) for c in r["checks"]])
EOF

The three rig cells are the rows whose check detail starts "could not read the phone". The summaries quoted in the tables above are the agent_said field of the unredacted run files on the rig. The check vocabulary is in phoneshell/bench/verify.py; the not-scored rules and the timed-out-cannot-pass guard are in Runner.run in phoneshell/bench/runner.py, both in github.com/Blomega/phoneshell. The McNemar figure uses mcnemar_exact(4, 0) from phoneshell/bench/stats.py, which returns 0.125.

What would prove this wrong

The claim is that an agent's own completion summary is biased upward, not merely noisy. The prediction, with a date: on the next paired sweep of at least 300 runs on this suite, before 31 December 2026, at least 40% of device-graded failures will carry a completion claim, and no more than 1% of device-graded passes will carry an admission of failure. A sweep where false admissions are as common as false claims falsifies the direction of the bias, and we will publish it.

The limit on everything above: one phone, first-party apps only, and summaries labelled by one reader. The labels for all 52 failures across both runs are short enough to recheck by hand, and a second reader disagreeing on more than five of them would weaken the count.