Four of six failures in our first trusted run were our own bugs, not the model's

The score said 85.4% of attempted tasks passed. Reading every failure against the agent's own transcript said something more useful: the instrument was wrong in three places, and no number of reruns would have found it.

The first run of this benchmark that we trusted scored 85.4% of attempted tasks. Then we read every failure against what the agent actually said and did on the phone. Of six failures, three were caused by our own checks, one was a task that could only be failed because of the owner's device, and one or two were the model's. The audit changed the interpretation of the run completely, and it took less time than the run did.

What the audit found, task by task

The method is the cheap part: for each failed task, put the agent's own transcript next to the device-state check that failed it, and ask which of the two is wrong. The agent's account is not evidence that it succeeded, because a model that believes it tapped Send will say it sent the message. It is excellent evidence about the harness, because a model that says "I created the reminder and the list went from 5 items to 6" and then fails a check is telling you the check is looking in the wrong place.

The last row is the one that matters most, because it was found after the first three were fixed. It is the fourth check of the same shape: an assertion on a control that only exists in one of the device's two legitimate states. On an empty phone the task is unpassable no matter what the agent does, and nothing in the result file says so.

Five ways this rig produced a number that was not true

Four of them made the number look better. That direction is not a coincidence: a harness fault usually removes work from the model or credits work it did not finish, and only rarely invents extra difficulty. Each one is now a guard in code rather than a note in a document.

The pattern behind all five is one distinction: the model tried and got it wrong and we could not ask the model are different facts. A benchmark that cannot tell them apart is not producing a noisy number, it is producing fiction, and the fiction is stable across reruns because the cause is in the harness.

This is not a local problem

The Agentic Benchmark Checklist (arXiv:2507.02825, first posted 3 July 2025, v5 on 7 August 2025) reports the same class of defect in published benchmarks: SWE-bench Verified uses insufficient test cases, TAU-bench counts empty responses as successful, and the authors put the resulting misestimation at up to 100% in relative terms. Applying the checklist to CVE-Bench reduced its performance overestimation by 33%. TAU-bench counting an empty response as a success is the same fault as our timed-out alarm task passing its checks with zero turns and no spend: a grader that reads the environment without asking whether the agent was ever in a position to change it.

The method we used by hand has since been automated. "Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks" (arXiv:2607.27518, 29 July 2026) runs scanners over benchmark transcripts looking for ground truth access, tool failure, guessing vulnerability and answer format ambiguity, and reports verified quality issues in five widely used benchmarks. Our four categories were different only because our environment is a phone: a check asserting on a control iOS does not draw, a courtesy feature pausing a measured run, a check that would not scroll, and a task whose outcome depended on how many Safari tabs the owner happened to have open.

What to do with this if you run an agent benchmark

Read the transcripts of your failures before you publish the number. Not a sample: all of them, at least once, at the point where the suite is small enough that this is possible. It is the cheapest quality control available and it is the only one that catches a check that is wrong in the same direction every time.

Then make the two failure modes structurally different in your results file. A boolean passed cannot express "we could not ask the model", and if the only place that fact lives is an exception message, the aggregate will silently absorb it. Our row carries skipped and skip_reason alongside passed, and the published rate is computed over scored tasks with the skip count printed next to it.

Three rules came out of the audit and each generalises beyond phones. "Not visible right now" and "not there" are different facts, so a check that refuses to scroll blames the model for the harness's impatience. Assert on the surface, not on a specific button, because button sets change between OS versions and an assertion on one makes the benchmark rot. And a measured run must take the device: a feature that politely yields to a human who is not there is a scoring bug during measurement.

A judgement, marked as one: the audit is worth more than the run. The score told us roughly what we already believed. The audit found four defects in the instrument, and three of them would have survived any number of reruns.

Check it yourself

The guards are all in one file, and the results are line-delimited JSON, so the arithmetic is inspectable without running anything on a phone.

git clone https://github.com/Blomega/phoneshell && cd phoneshell

# the three guards, in order of the diagram above
grep -n "class UsageLimitReached" -A 8 phoneshell/bench/runner.py
grep -n "class RigUnavailable" -A 8 phoneshell/bench/runner.py
grep -n "result.passed = passed and not timed_out" -B 6 phoneshell/bench/runner.py

# scoring excludes skipped tasks and retired tasks, and keeps the latest attempt only
grep -n "def score" -A 30 phoneshell/bench/runner.py

# on your own results file: how many rows were never scored, and why
jq -r 'select(.skipped==true) | [.task_id,.model,.skip_reason] | @tsv' runtime/bench/latest.jsonl

# the audit itself: every failure next to what the agent said about it
jq -r 'select(.passed==false and .skipped!=true) | [.task_id,.turns,.seconds,.agent_said] | @tsv' \
  runtime/bench/latest.jsonl

The last command is the whole method. If the fourth column explains the task and the check still failed it, the defect is yours.

What would prove this wrong

• Take our published task set, run it against a model on your own iPhone, and audit every failure the same way. If fewer than one in four of your failures turns out to be a harness defect, our claim that this is the normal ratio for a young benchmark is wrong, and we would rather know. We predict a rate above one in four for any device benchmark under a year old, through 30 June 2027.
• Show a results file where skipped and passed==false aggregate to the same published rate. That would mean the distinction we built the guards around does not move the number in practice, and the guards are ceremony.
• The claim we are least sure of: that four of the five fabrication modes made the score look better. That is an observation over five incidents on one rig, which is an anecdote with a pattern, not a measured asymmetry. A larger sample from any device benchmark would settle it.