analysis 17 October 2026 8 min read

ProBench's best model scores 45.6% when the final screen is enough and 27.9% when a step in between must be checked; every one of phoneshell's 80 checks reads only the final screen

ProBench splits its mobile tasks into two kinds: ones a final screenshot can grade, and ones where the thing that mattered, a sort order applied, a filter set, a wrong turn taken, never shows up in the last frame at all. Its best model loses 17.7 points moving from the first kind to the second. phoneshell's own check code, read directly, never looks anywhere but the last frame.

ProBench (arXiv:2511.09157), submitted 12 November 2025 by Leyang Yang, Ziwei Wang and five coauthors from Zhejiang University and Ant Group, and published at AAAI 2026, starts from a specific complaint about how mobile GUI agents get graded: "current GUI agent benchmarks judge task completion only by the final screen state, but important information can appear in intermediate steps." Their example is concrete: finding the highest-rated sushi restaurant in Tokyo requires locating Tokyo, sorting results by rating, and picking a sushi place specifically, three decisions, and the final screen, a restaurant's menu, looks identical whether the agent actually sorted by rating or just tapped the first result it saw. ProBench keeps the older final-screen checks, which it calls State-related Tasks (ST), and adds over 200 tasks total with a second category, Process-related Tasks (PT), graded by a dedicated Process Provider that captures what happened along the way.

Abstract technical illustration on a dark grid: a single bright endpoint node reachable by two different glowing paths, one short and direct representing a final-state check, one long and winding through several dimmer intermediate nodes representing a process check, with the winding path's nodes rendered with a visible but fainter trail that the short path lacks, cool blues and teals with one amber accent on the winding path

Every evaluated model drops on process tasks, none rises

Table 3 of the paper scores ten models on both task types. Gemini 2.5 Pro is the best model overall: 45.6% on State-related tasks, 27.9% on Process-related tasks, an average of 40.1% and the largest single-model drop in absolute points, 17.7. Qwen2.5-VL-72B shows the same direction at a lower level, 40.9% ST against 27.9% PT. Three of the ten models, GPT-4o, Claude 4 Sonnet and UI-R1-E-3B, score 0.0% on both task types in this evaluation, which the paper attributes to the models failing to complete even the state-related portion of these specific tasks. Every model in the table that scores above 0% on ST also scores lower on PT than on ST; none of the ten reverses that direction. The paper traces failures to two causes directly: agents that ignore a stated constraint rather than execute it (its example is a goggles search where Gemini 2.5 Pro skipped an implicit price-sort requirement and simply scrolled), and agents that pass a process check only because the step was functionally unavoidable, not because the agent recognized it as a requirement, which the authors flag explicitly as not proof of real capability.

Model	State-related (ST)	Process-related (PT)	Average	Source
Gemini 2.5 Pro	45.6%	27.9%	40.1%	arXiv:2511.09157 Table 3
Qwen2.5-VL-72B	40.9%	27.9%	36.9%	arXiv:2511.09157 Table 3
Qwen2.5-VL-32B	18.8%	11.8%	16.6%	arXiv:2511.09157 Table 3
UI-TARS-1.5-7B	11.4%	2.9%	8.8%	arXiv:2511.09157 Table 3
GUI-R1-3B	12.1%	0.0%	8.3%	arXiv:2511.09157 Table 3
Qwen2.5-VL-7B	6.7%	1.5%	5.1%	arXiv:2511.09157 Table 3
GPT-4o	0.0%	0.0%	0.0%	arXiv:2511.09157 Table 3

phoneshell's checks run once, after the agent is already finished

phoneshell's (github.com/Blomega/phoneshell) grading code lives in phoneshell/bench/verify.py and phoneshell/bench/runner.py. The runner's own docstring states its loop as "setup (no model) -> hand the instruction to the agent -> check the phone," and the actual call, run_all(self.phone, task.checks, said=result.agent_said), happens after run_agent returns, not during it. There are five check kinds across the 80 environment files: foreground_app reads which app bundle is currently frontmost, element_exists and element_exists_anywhere read the current accessibility tree (the second one scrolls first), regex_on_screen pattern-matches whatever text is visible right now, and answer_contains, the one documented exception, grades the model's own stated answer rather than phone state, reserved for tasks whose deliverable is information rather than a state change, such as reading a word out of an image. Every one of the other four reads the phone exactly once, at the moment run_all is called, and none of them inspects what the screen looked like three steps earlier.

That design is deliberate and stated, not an oversight: verify.py's own docstring reads "every check reads the phone directly rather than believing the agent's account of itself," which is a different, narrower promise than ProBench's. phoneshell's checks guarantee that whatever is claimed about the final state is true. They make no claim at all about whether the agent took the specified route to get there. A phoneshell task whose environment file does not happen to require a visible trace of sorting, filtering, or an intermediate choice, which is most of them, cannot distinguish an agent that did the work from one that got lucky, the exact gap ProBench's Process-related Tasks were built to close.

What it means for anyone grading a phone agent honestly

A benchmark that only reads final state is not wrong, it is answering a narrower question than it may appear to. ProBench's own numbers show that narrower question has an easier, more forgiving answer: every one of its ten models scored higher on ST than on PT, often by double.
Adding process-aware grading to a harness like phoneshell's would need a new check kind that samples the accessibility tree or a screenshot partway through the agent's turns, not at the end, which the current run_all call, by design, never does.
The three zero-scoring models (GPT-4o, Claude 4 Sonnet, UI-R1-E-3B on this evaluation) are a reminder that a process-aware check only matters once a model clears the state-related bar first; it adds a second, harder filter on top of the first one, not a replacement for it.

Check it yourself

# confirm phoneshell's checks run once, after the agent finishes
grep -n "run_all\|run_agent" phoneshell/bench/runner.py
sed -n '1,20p' phoneshell/bench/verify.py

# the five check kinds and what each one reads
grep -n 'if kind ==\|if check.kind ==' phoneshell/bench/verify.py

# ProBench's own task split and Process Provider
curl -s https://arxiv.org/abs/2511.09157 | grep -i "process-related\|state-related\|Process Provider"
# full tables: https://arxiv.org/html/2511.09157v1 (Table 3, error analysis section)

What would prove this wrong

We claim phoneshell's checks read device state exactly once, after the agent's turn ends, and that none of its five check kinds inspects an intermediate screen. That is wrong the moment a future commit to phoneshell/bench/runner.py calls a check before run_agent returns, or adds a check kind that samples the tree mid-task. We also report ProBench's Gemini 2.5 Pro gap as 17.7 points from Table 3; that figure is wrong if a later revision of the paper restates the table with different numbers. We will check both by 31 March 2027.

Sources

Yang, Wang, et al., ProBench: Benchmarking GUI Agents with Accurate Process Information, arXiv:2511.09157, submitted 12 November 2025, AAAI 2026, read 17 October 2026.
Full tables and error analysis: arxiv.org/html/2511.09157v1.
Blolabel, phoneshell: phoneshell/bench/verify.py, phoneshell/bench/runner.py, read 17 October 2026.