GMA's biggest realistic-tier jump is captions, not more frames: 11.0 to 34.6 at the same two screenshots; phoneshell already serializes its screen as text, never the picture

Eight frontier models tested on GMA's hardest tier either score 0.00 or fail to clear 20%. The paper's own harness ablation, run on one of those models, found that going from 2 screenshots to 16 barely moved its hardest-tier score, but adding a text caption to the same 2 screenshots nearly tripled it, from 11.00 to 34.60. phoneshell already sends a text description of every screen on every step. It has never captioned the picture that goes with it.

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios (arXiv:2608.27477, Zhu, Gao, Fan, Zeng, Wu, Li, Xu, Li, Yan and Liu, submitted 21 August 2026) tests 8 frontier models, including GPT-5.5, GPT-5.6 Sol, Claude Sonnet 4.6, Claude Opus 4.7, Gemini 3.5 Flash, Doubao-Seed-2.1-pro, Qwen3.7-Plus and Qwen3.8-Max, on 300 tasks across 7 open-source apps, split into four difficulty tiers: Atomic, Compositional, Cross-Application, and Realistic. The paper's own line is blunt: "No model completes more than 20% of these tasks perfectly." Its Table 1 shows why. Every model that scores above 80% on Atomic tasks lands at 0.00 to 20.00 on Realistic ones, the tier meant to resemble how someone actually uses a phone rather than a scripted, single-intent request.

Where every model falls apart, and by how much

GMA's four tiers are cumulative in difficulty: Atomic is a single UI action, Compositional chains a few inside one app, Cross-Application spans two or more apps, and Realistic adds the ambiguity and multi-step planning of an actual use case inside the same app set. The drop from Atomic to Realistic is not gradual. Qwen3.8-Max, the strongest model on the easy end at 93.15% Atomic, falls to 10.00% on Realistic, an 83.15-point drop. Four of the eight models, Qwen3.7-Plus, Gemini 3.5 Flash, GPT-5.5 and Claude Sonnet 4.6, score exactly 0.00% on Realistic. Doubao-Seed-2.1-pro posts the best Realistic score in the table at 20.00%, still one task in five.

Source: arXiv:2608.27477, Table 1, success rate (%) by tier, read 29 October 2026.

What the paper's own harness ablation changed, and what it did not

GMA's authors ran a separate ablation on Qwen3.7-Plus alone, varying how much screen history the agent's context carries and in what form, measured against the same Realistic Average. Scaling the image count on its own does almost nothing: 2 images with a thought step scores a 11.00 Realistic Average, 4 images scores 12.86, 8 images scores 21.44, and 16 images scores 23.81, a slow climb that costs four to eight times the image tokens for roughly double the score. The paper's own line: "explicitly maintaining task state can provide useful information beyond simply retaining raw interaction history, particularly when the available visual history is limited." The single largest jump in the whole table is not from any of those: keeping the image count fixed at 2 and adding text captions alongside them took the Realistic Average from 11.00 to 34.60, more than the 16-image run achieved with eight times the pictures.

Source: arXiv:2608.27477, Table 2, read 29 October 2026. Ablation run on Qwen3.7-Plus only; the paper does not report this ablation for the other seven models.

What phoneshell's own observation step already builds, and never adds

phoneshell's agent/observation.py states the reasoning for its own context format in its opening docstring: hybrid tree-plus-image perception beats either channel alone in the literature it cites (53.7% versus 35.2% tree-only versus 15.6% pixels-only), and it serializes the accessibility tree as TSV rather than JSON specifically to save tokens, "30-40% more tokens for zero extra information" is how the file describes the alternative. Every step phoneshell sends a model already carries two things GMA's caption condition approximates separately: a textual description of what is on screen (the TSV) and the screenshot itself, scaled down because "a 512px-wide frame is ~760 tokens and loses nothing that matters at tap granularity," per the same file. What phoneshell has never done is caption the image independently of the tree serialization, the specific intervention GMA's ablation isolated as the one that moved the Realistic score the most.

The distinction matters because phoneshell's TSV already tells a model what the accessibility tree reports is on screen, structurally. A caption, in GMA's sense, would describe what the picture itself shows, which can disagree with the tree when a view is custom-drawn, a WebView, or stale. agent/observation.py names exactly this risk in its own OPAQUE_TYPES set: WebView, Map, Other, ScrollView and Image, container types the file's own comment says mean "the tree is probably lying about what is here." GMA's caption condition is, in effect, a second read of the screen that does not trust the tree either, run through the same model as a separate step rather than skipped.

What it means for a harness that already picked tree-plus-image

• GMA's own caveat applies here too: this ablation ran on one model, Qwen3.7-Plus, on one benchmark's Realistic tier, with one caption-generation setup the paper does not fully specify in the parts we could read. A gain this size on one model is a strong signal to test, not a result to assume transfers to GPT-5.6 Sol or Claude Opus 4.7, both of which outscore Qwen3.7-Plus on GMA's Realistic tier without any caption step at all.
• phoneshell's own rationale for TSV over JSON, and for a 512px frame over full resolution, is about not wasting tokens on information the tree or a smaller image already carries. A caption step adds tokens and a model call on every step; the honest comparison phoneshell would need is whether a caption changes which of its 80 tasks fail, not just whether it changes the token bill.
• The sharpest test case for phoneshell would be exactly the screens its own OPAQUE_TYPES set already flags as untrustworthy, WebView and custom-drawn views, where the tree and the picture are most likely to disagree and a caption would carry information the TSV cannot.

Check it yourself

# phoneshell's own reasoning for TSV over JSON and image scaling
sed -n '1,14p' phoneshell/agent/observation.py

# the container types phoneshell's own file says the tree may be lying about
grep -n "OPAQUE_TYPES" -A3 phoneshell/agent/observation.py

# confirm no caption-generation call exists anywhere in the agent loop
grep -rn "caption" --include="*.py" phoneshell/ || echo "no match"

# GMA's own ablation table and tier collapse, read from the source
curl -s https://arxiv.org/html/2608.27477v1 | grep -iE "Realistic|caption|Table 1|Table 2"

What would prove this wrong

We claim phoneshell has no code path, anywhere in its agent loop, that generates a text caption of a screenshot independent of the accessibility tree serialization it already builds. That is wrong the moment a caption call turns up in the repository, and we encourage anyone to check the grep above themselves. We also claim GMA's own data shows captions, not image count, drove its largest ablation gain on Qwen3.7-Plus: that one model, one benchmark result does not establish that captioning would move phoneshell's own 80-task pass rate, which this site has not tested with or without captions. We will treat a repository change adding captioning, or a run of phoneshell's own benchmark with captions isolated as the only variable, as grounds to revisit this piece by 30 April 2027.