A3's MLLM reward model roughly doubles average success from 19.0% to 36.5% across ten agents; phoneshell's checks run with no judge model in the loop at all

A3 scores the same 100 live-app tasks two ways, and the two ways do not agree: a strict pass/fail success rate, and a more forgiving essential-state score produced by an MLLM acting as a reward model. The reward model's own accuracy is reported and is not perfect. phoneshell's checks never ask a model anything; they read the accessibility tree and the screen directly, every time.

A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation (arXiv:2501.01149), submitted 2 January 2025 by Yuxiang Chai and 14 coauthors, revised most recently on 12 January 2026 (v3) ahead of ACL 2026 Findings, starts from a complaint about where mobile benchmarks run: AndroidControl grades against static frames, and AndroidWorld grades inside offline, non-live apps, so neither captures how an agent behaves against a real, changing app. A3 fixes that by running 100 tasks across 20 widely used, live online apps, and then grades each run two different ways. A strict success rate (SR) counts only a clean, complete pass. An essential-state score (A3 calls it ESAR) comes from a purpose-built MLLM reward model, A3RM, which scans a sliding window over the agent's trajectory and judges, segment by segment, whether any of a task's defined "essential states," intermediate milestones a human labeler identified in advance, were actually reached, crediting progress a strict pass/fail check would discard entirely.

The judge's score is roughly double the strict score, for nearly every agent

Table 3, graded by A3RM, lists ten agents. The pattern holds across almost all of them: UI-TARS-1.5 scores 12% SR and 28.2% ESAR, UI-Genie scores 13% SR and 32.1% ESAR, GUI-OWL scores 14% SR and 32.0% ESAR, InfiGUI-R1, the best open-source single-model agent, scores 27% SR and 52.1% ESAR, and the best agent overall, T3A paired with Gemini-2.5-pro, scores 53% SR and 66.4% ESAR. Averaged across all ten agents in the table, mean SR is 19.0% and mean ESAR is 36.5%, just under double. A second table, Table 6, regrades the same trajectories with a different judge, Gemini-2.5-pro instead of A3RM, and the gap persists with similar or larger size: T3A with Gemini-2.5-pro reaches 58% SR and 69.3% ESAR under that judge. The paper's own validation of A3RM against held-out human labels reports, at the essential-state level, 95.7% precision, 94.8% recall, and 96.6% accuracy, and at the task level, 96.0% precision, 96.0% recall, and 98.0% accuracy, adding, in its own words, that "A3RM is more aligned to human judgment and Gemini suffers from the low precision situation." No inter-annotator agreement figure for the human labels themselves is reported.

phoneshell has no judge model to be 95.7% accurate, or any other number

phoneshell's (github.com/Blomega/phoneshell) grading, in phoneshell/bench/verify.py, states its own rule in its module docstring: "Every check reads the phone directly rather than believing the agent's account of itself... Where a check can be made against real device state rather than against pixels, it is: state is harder to fake and does not move when Apple redesigns a screen." The five check kinds, foreground_app, element_exists, element_exists_anywhere, regex_on_screen, and answer_contains, are deterministic functions over the accessibility tree, screen text, or the model's stated answer against a known string. None of them calls an LLM, vision model, or any other model to interpret what it sees. There is consequently no analogue to A3RM's reported precision or recall anywhere in phoneshell, not because the number would be good or bad, but because there is no judge in the loop to measure in the first place.

That trade runs in both directions, and A3's own numbers show what phoneshell's design gives up. phoneshell's checks cannot award partial credit the way A3RM does: a phoneshell task is pass or fail against whatever its environment file's checks state, with no equivalent of ESAR's progress-aware scoring for a task that got partway there. A3's checks, in exchange for that flexibility, inherit whatever error the judge itself carries, 4.3% of state-level judgments disagreeing with a human label by A3RM's own reported precision, and a second, different judge (Gemini-2.5-pro) producing yet another, measurably different set of scores on the same trajectories in Table 6. phoneshell's bet is that a narrower, strictly pass/fail signal that is never itself a probabilistic judgment is worth more than partial credit that depends on how good this run's judge happens to be.

What it means for anyone choosing how to grade a phone agent

• A benchmark's grading method is itself a result that needs its own accuracy number. A3 reports one (95.7% precision, 94.8% recall at the state level); many benchmarks that use an LLM judge do not, which makes their reported success rates harder to compare against a deterministic one like phoneshell's.
• The near-universal direction of the gap, every one of A3's ten agents scores higher under ESAR than under SR, means a reader comparing an ESAR number from A3 against a strict SR number from any other benchmark, phoneshell's leaderboard included, is comparing two different kinds of measurement, not two agents.
• Using two different judges on the same data (A3RM in Table 3, Gemini-2.5-pro in Table 6) and getting two different ESAR numbers for the same agent is itself evidence for phoneshell's design choice: a judge-based score is specific to the judge used to produce it, in a way a tree-read check is not.

Check it yourself

# confirm phoneshell's checks never call a model
grep -n "if check.kind ==\|if kind ==" phoneshell/bench/verify.py
sed -n '1,10p' phoneshell/bench/verify.py

# look for any LLM/vision-judge call anywhere in the grading path
grep -rn "openai\|anthropic\|gemini\|judge\|reward_model" phoneshell/bench/verify.py phoneshell/bench/runner.py

# A3's own two-judge tables and reward-model validation
curl -s https://arxiv.org/abs/2501.01149 | grep -i "A3RM\|essential-state\|precision"
# full tables: https://arxiv.org/html/2501.01149v3 (Table 3, Table 6, Table 2 validation)

What would prove this wrong

We claim none of phoneshell's five check kinds calls any model, LLM or otherwise, to judge a run; every one is a deterministic read of the accessibility tree, screen text, or a literal string match. That is wrong the moment a future commit to phoneshell/bench/verify.py adds a check kind that asks a model to interpret a screen or trajectory. We also report the ten-agent mean gap between SR and ESAR in A3's Table 3 as 19.0% to 36.5%; that computed average is wrong if a later revision of the paper restates Table 3 with different per-agent figures. We will check both by 31 March 2027.

Sources

• Chai, Tang, et al., A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation, arXiv:2501.01149, submitted 2 January 2025, revised 12 January 2026 (v3), ACL 2026 Findings, read 17 October 2026.
• Full tables and A3RM validation: arxiv.org/html/2501.01149v3 (Table 2 reward-model validation, Table 3 A3RM-judged results, Table 6 Gemini-2.5-pro-judged results).
• Blolabel, phoneshell: phoneshell/bench/verify.py, read 17 October 2026.