A3 rejects device state as "not suitable" and grades 100 tasks from screenshots alone; phoneshell rejects screenshots the same way and grades from device state

A3's own paper states that function-based checks against device internals are "not suitable for this task of dynamic, constantly updating apps," so its reward model judges every task from a sliding window of screenshots, never from the phone's actual state. phoneshell's bench/verify.py states the mirror-image rule for the same category of app: a check reads device state wherever it can, because "state is harder to fake and does not move when Apple redesigns a screen." Both projects name the same problem and pick opposite sides.

A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluation (arXiv:2501.01149, Chai, Tang, Xiao, Lin, Li, Zhang, Liu, Zhao, Liu, Wang, Ren, Han, Zhang, Huang, Li; first submitted 2 January 2025, latest revision 12 January 2026) builds a 100-task benchmark from 20 widely used, dynamic Google Play apps across 20 categories, and its central contribution is how it decides whether an agent actually finished a task. An "essential state" is defined in the paper as "a critical, observable semantic milestone that must be achieved for a task to be considered successful," produced by a three-stage human protocol: two operators independently complete each task by different strategies to surface varied trajectories, the operators jointly propose essential states that are visually verifiable and sufficient to cover the task's logic, and an independent auditor checks the result. At evaluation time, a sliding window of recorded frames moves through the agent's run, and at each window position a multimodal reward model, A3RM, judges whether any essential state is visible in that segment. The paper is explicit about what this method deliberately does not do: it never inspects the device's accessibility tree, view hierarchy, or any other app-internal state, calling that approach unsuitable for apps whose UI and backend content change constantly.

What judging from pixels alone scores, by agent

Table 3 of the paper, evaluated by A3RM, reports two figures per agent: SR, the fraction of tasks where every essential state was found, and ESAR, essential-state achievement rate, the fraction of individual essential states found across all tasks regardless of whether the full task completed. The gap between the two numbers is the paper's own evidence that single-state credit and whole-task credit diverge sharply: T3A paired with Gemini-2.5-pro reaches 53% SR against 66.4% ESAR, and InfiGUI-R1, the best fully open single-model agent, reaches 27% SR against 52.1% ESAR, meaning it clears roughly half of the individual milestones in a task about twice as often as it clears every milestone the task has. Table 6, a separate evaluation of the same runs using Gemini-2.5-pro directly as the judge instead of the trained A3RM, gives different SR numbers for several agents, which the paper reports side by side without resolving which judge is more accurate.

Source: arXiv:2501.01149v3, Table 3 (A3RM-evaluated) and Table 6 (Gemini-2.5-pro-evaluated), read 30 October 2026.

Why phoneshell's checker makes the opposite bet

phoneshell faces the same category of app, dynamic, real, closed-source consumer software, and its own module docstring in bench/verify.py states the reasoning directly: "Every check reads the phone directly rather than believing the agent's account of itself... Where a check can be made against real device state rather than against pixels, it is: state is harder to fake and does not move when Apple redesigns a screen." Its checks dispatch against a live accessibility tree, a foreground app identifier, or a UI element's actual position and value, read through WebDriverAgent. The one exception the file names explicitly is the answer_contains check, used only for tasks whose deliverable is information rather than a state change, where there is nothing on the device to inspect, so the model's stated answer is scanned instead, against a page phoneshell hosts itself so the ground truth cannot drift. That single carve-out is the full extent to which phoneshell's grading ever resembles A3RM's screenshot-only judgment: everywhere a state read is possible, phoneshell takes it, and A3 takes the opposite path by design, arguing that state reads do not generalize across a diverse app set the way a visual judge does.

What it means for choosing a verification method

• A3's rejection of device state is a scale argument: 20 apps across 20 categories, built and maintained by different companies, cannot be assumed to expose a stable or even accessible internal state, so a visual judge is the only method that transfers across all of them without per-app engineering.
• phoneshell's acceptance of device state is a depth argument: within one phone's set of real apps, deliberately curated and re-verified when they change, a tree read is available often enough, and reliable enough once verify.py's own consistency gate confirms the label is actually drawn where claimed, that paying the engineering cost per app is worth not depending on a judge model's visual accuracy at all.
• The two are not actually in conflict as a general claim, they are each correct for the regime they were built for. A system trying to cover hundreds of unfamiliar third-party apps should expect to need something closer to A3RM; a system holding a fixed, maintained set of real apps on one device should expect device-state checks to pay for themselves, as phoneshell's own 153 assertions across 80 tasks, 133 of them already a direct state read according to this site's own prior measurement, suggest it has.

Check it yourself

# phoneshell's own stated reasoning for preferring device state
sed -n '1,10p' phoneshell/bench/verify.py

# the one exception: grading from the model's stated answer, not device state
grep -n "answer_contains" -A10 phoneshell/bench/verify.py

# every other check kind dispatches against live device state
grep -n "^def _dispatch" -A 15 phoneshell/bench/verify.py

# A3's own stated rejection of function-based, device-state checks
curl -s https://arxiv.org/html/2501.01149v3 | grep -iE "not suitable|essential state|A3RM"

# A3's Table 3 and Table 6 results, as quoted above
curl -s https://arxiv.org/html/2501.01149v3 | grep -iE "Table 3|Table 6|ESAR"

What would prove this wrong

We claim A3's paper explicitly calls function-based, device-state verification unsuitable for its benchmark and that A3RM's judgment is based on screenshots alone with no accessibility-tree or view-hierarchy input anywhere in the pipeline; that is checkable against the paper's section 3.3 and would be false if a later revision adds any state-reading component. We also claim phoneshell's checker reads device state for every check kind except answer_contains; that is checkable directly against bench/verify.py's dispatch table and would be false the moment a screenshot-only judge is added there. Neither claim says one project's choice is correct in general, only that the two made opposite calls for a similar-sounding problem; we will revisit this piece if phoneshell ever adds a vision-judged check kind, or if a later version of A3 adds device-state verification, by 30 April 2027.