MobiBench: a raw screenshot grounds Android actions at 4.72%, the accessibility tree at 27.17%; phoneshell's vision question was never about which the agent acts from

Two "vision versus the tree" results from two different projects, pointing in what looks like opposite directions until you notice they are not measuring the same job. One is about choosing where to tap. The other is about whether the referee needs to watch the screen.

MobiBench, a modular offline benchmark for mobile GUI agents from KAIST, Sungkyunkwan University, Korea University and Fluiz, scores GPT-4.1 at 27.17% task success reading an Android accessibility tree as linearized HTML-style text, and 4.72% reading a raw screenshot with no text channel at all, a gap of more than five times. A hybrid feeding both the screenshot and the tree reaches 31.50%, the best of the three. phoneshell's own headline vision result, measured on a physical iPhone, is 100% success with a screenshot against 0% without it, on pages built so the answer exists only as pixels. Read quickly, these look like the same debate reaching opposite conclusions: one says text beats pixels, the other says pixels beat nothing. Read carefully, MobiBench is scoring which input helps an agent decide its next tap. phoneshell's probe is scoring whether the device-state checker needs to see the screen to know what happened. Those are different jobs, and a single phrase, "vision versus the accessibility tree," is being asked to cover both.

What changed, and when

MobiBench was posted to arXiv as "Modular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents" (arXiv:2512.12634), with v1 on 14 December 2025 and a revised v2 on 8 January 2026, and its code and dataset are published at github.com/fclab-skku/Mobi-Bench. Its central dataset holds 508 tasks across 66 apps, 4,173 screenshots and 12,339 annotated actions, built so that more than one action can be marked correct at each step rather than penalizing an agent for a valid alternative a single-path annotation never recorded. A 105-task subset is used to compare directly against AndroidWorld, the standard online Android benchmark.

phoneshell's render-stratum result comes from experiment 2, pre-registered in docs/EXPERIMENT-2.md before the run: twenty tasks, three vision-capable models, each run with and without the screenshot, with ten tasks engineered so the answer is a word rendered into an image with no alt text, verified absent from the accessibility tree on 10 of 10 pages. Neither study is newer than the other in a way that matters here; they were simply never asking each other's question.

Two jobs, measured separately

What this means for someone building or reading about these agents

When a benchmark reports "vision helped" or "vision didn't help," the first question worth asking is which of the two jobs it tested. MobiBench's result is a caution against assuming a screenshot alone is a sufficient channel for an agent to act from on Android: naming beats guessing coordinates by a wide margin, and a developer choosing a parser for an action-taking agent should default to structured text and treat raw pixels as a supplement, not a replacement, matching the 2.7x hybrid gain MobiBench reports for the weaker model. phoneshell's result is a caution in the opposite direction, aimed at whoever is deciding whether a grading or verification step needs to look at the screen: the answer there is not "rarely" or "sometimes," it is "exactly when the fact you need is not already exposed to assistive technology," which is knowable per screen in advance with a tool like render_gap.py, before spending anything on a vision call. Neither study tells you the other's answer. A team that reads MobiBench and concludes vision is broadly weak, or reads phoneshell and concludes vision is broadly essential, has generalized past what either one measured.

MobiBench's other headline number is adjacent context worth carrying over separately: scoring with a single annotated correct path, the common offline-benchmark design, underestimated its own multi-path, human-validated score by up to 16.09 percentage points, a 49.9% relative gap (16.18% single-path versus 32.27% human-agreement TSR in its own comparison table). That is not a vision-versus-tree finding; it is a caution about crediting only one correct action per step at all, text or pixels, and belongs with phoneshell's own repeated point that a benchmark's scoring method can manufacture a result the model never earned.

Check it yourself

git clone https://github.com/fclab-skku/Mobi-Bench && cd Mobi-Bench   # MobiBench's 508 tasks, 66 apps
# Table 5 in the paper (arXiv:2512.12634) holds the per-parser action-accuracy and TSR numbers quoted above

git clone https://github.com/Blomega/phoneshell && cd phoneshell
cat docs/EXPERIMENT-2.md                 # phoneshell's pre-registered grading probe
grep -A6 "render stratum" FINDINGS.md    # section 26, 30/30 vs 0/30

What would prove this wrong

• We predict that if MobiBench's exact three-configuration comparison (text-only, screenshot-only, hybrid) were run on phoneshell's own render-stratum pages, where the tree carries nothing by construction, the text-only configuration would score near zero rather than 27.17%, because there would be no element names to extract. MobiBench's 4.72% screenshot-only number comes from Android apps where usable text existed in the tree and pixels still lost to it; that is a different situation from a page where the tree has nothing to offer in the first place. Untested by either project as of this writing, falsifiable by whoever runs it, through 31 December 2027.
• If a future MobiBench revision reports its per-task breakdown of which specific tasks the hybrid configuration recovers over text-only, and none of those tasks resemble phoneshell's render stratum (pixel-only facts with no text analogue anywhere on screen), that would weaken the resonance we draw between MobiBench's hybrid gain and phoneshell's own vision-decisive cases, and we would want to revise this comparison accordingly.