The best mobile-agent judge is 90.9% accurate, and the gaps it is used to report are smaller than that

Three of the four mobile-agent benchmarks published in 2026 hand the pass decision to a model, and each reports agreement with humans rather than with the device. Here is what that buys, and what it costs.

The best-performing judge in a head-to-head benchmark of mobile-agent evaluators scored 90.9% accuracy against human labels (Wang et al., arXiv:2608.11434, 11 August 2026, 931 annotated trajectories). Two human annotators agree with each other 88.4% of the time on the same success label. Both numbers are lower than they sound, because the differences these judges are used to report are four points and smaller. A 133-task benchmark graded by a 90.9% judge carries roughly 12 mislabelled tasks; a 4.0 point gap between two models is about 5.

Four benchmarks, four different deciders

Mobile and computer-use benchmarks published in 2026 do not disagree about the tasks nearly as much as they disagree about who decides a task was done.

Three of those five hand the decision to a model. The reason is not laziness. AndroidDaily states the constraint plainly: closed-source applications do not expose internal states, so there is nothing for a programmatic checker to read, and the choice is a model judge or no benchmark at all on the 94 apps people actually use.

The error is bigger than the result

Wang and colleagues replayed 931 human-annotated trajectories from SPA-Bench, AndroidWorld, A3, AndroidArena, B-MoCA and AndroidLab, covering 289 tasks across 68 apps, and ran six judge methods over five backends. The simple baseline judge won: 90.9% with Gemini 3 Flash, 90.8% with GPT-5-mini, 85.7% with Qwen2.5-VL-72B. Human annotators agree with each other 88.4% of the time on the success label.

Judge error partly cancels between two models graded by the same judge, but only if the errors are symmetric across the arms. The judge benchmark's most useful result says not to assume it: the failure profiles are "qualitatively opposite" depending on the backend. GPT-based judges skew false-negative, with last-frame anchoring (12 cases), unfamiliar success states (9) and invisible state changes (7). Qwen-based judges skew false-positive, with constraint violations and partial completions (27 cases each) and surface UI matches (17). A judge that concludes success from a screen that merely looks right will not make that error evenly when one arm sees screenshots and the other does not.

That is a judgement, marked as one: we have not run an ablation that measures asymmetry directly, and neither has any paper cited here.

What a checker buys, and what it costs

Our own suite takes the other option and pays for it in coverage. Eighty tasks, 16 bundle ids, all first-party Apple apps, because a deterministic check only works where the phone will tell you the answer. There is no food-delivery task in it. AndroidDaily has 94 apps and 350 tasks precisely because GRADE does not need the app to expose anything.

The checker has failure modes too, and they are ours rather than the model's. Seven of our 80 tasks assert nothing but the foreground bundle id, and three of those ask for a sequence whose first leg already satisfies the assertion. Twenty tasks grade the model's sentence rather than the phone, which is the right call for "what word is in that picture" and would be indefensible for anything with a state change in it. A checker cannot misread a screenshot. It can absolutely assert the wrong thing, and when it does, it does so identically ten thousand times.

The practical rule we now use: report the evaluator with the result, the same way a paper reports its test. "GPT-5.4 scored 39.8%" is not a fact about GPT-5.4. It is a fact about GPT-5.4 under a GPT-5.4-Mini judge, on a simulator, with seeded data, and the difference between it and the next model needs to clear the judge's error before it means anything.

Check it yourself

# the judge benchmark's own numbers
open https://arxiv.org/abs/2608.11434     # 931 trajectories, 6 judges x 5 backends
open https://arxiv.org/abs/2605.27761     # GRADE, 87.37% agreement, 879 sessions
open https://arxiv.org/abs/2606.09764     # iOSWorld, GPT-5.4-Mini judge, kappa 0.77

# the checker side, counted from the task files
git clone https://github.com/Blomega/phoneshell && cd phoneshell
python3 -c "import yaml,glob,collections; c=collections.Counter(
  k['kind'] for f in glob.glob('environments/*.yaml') for k in yaml.safe_load(open(f))['checks']);
  print(c.most_common())"
sed -n '1,12p' phoneshell/bench/runner.py

What would prove this wrong

The claim is that published model gaps on judge-graded mobile benchmarks are inside the judge's error. It is testable without new tasks: take one benchmark, grade the same trajectories with two different judge backends, and compare the rankings. If the ordering of the top three models is stable across a GPT-based and a Qwen-based judge on iOSWorld or AndroidDaily, then judge error is cancelling and the rankings survive. If the ordering moves, they do not. We will run that comparison on our own 80 tasks against a judge by 30 November 2026 and publish both orderings, whichever way it falls.