Structured state tracking lifts GMA's Qwen3.7-Plus from 49.3% to 54.3% success; phoneshell's own scaffold prompt tracks no state at all

GMA holds the model fixed and changes only the harness around it: how many screenshots it keeps, whether it writes down its own state. Both levers move success rate on their own. phoneshell already publishes its own harness, the scaffold prompt a model sees before a benchmark run, and that prompt does neither of the two things GMA found to matter.

"Benchmarking General Mobile Assistants in Challenging Real-World Scenarios" (arXiv:2608.27477), submitted 21 August 2026 by Yiqi Zhu and nine coauthors, introduces GMA, a benchmark of 300 tasks across seven apps built from open-source projects, split into four difficulty tiers from atomic single actions to realistic multi-step workflows. Eight frontier models were run against it, and the paper's headline finding is unsurprising on its face, performance drops as task complexity rises, but its most useful result is a controlled ablation isolating what part of that drop belongs to the model and what part belongs to the harness wrapped around it. Holding Qwen3.7-Plus fixed, the authors vary only two things: how many past screenshots the agent is allowed to keep in context, and whether it is given a structured place to write down what it currently knows about app state. Both move the number on their own.

Two harness levers, same model, two different scores

Table 2 of the paper holds Qwen3.7-Plus's reasoning fixed and varies only how many of its own past screenshots it is allowed to keep: with one retained image and a thought field but no caption, overall success is 43.33%. Doubling to two images with the same setup raises it to 49.33%. Going further, to four, eight, or sixteen images, raises it again to 51.00%, then plateaus at 52.00% for both eight and sixteen, the paper's evidence that this particular lever saturates rather than climbing indefinitely. A second table in the same ablation, Table 3, holds image count fixed and varies whether the agent gets a place to track state explicitly: no state-tracking design scores 49.33%, a free-form text field for state scores 50.33%, and a structured, schema-shaped state field scores 54.33%, a 5-point gain over no tracking at all from structure alone. The same structured-versus-none comparison on GPT-5.6 Sol moves from 62.67% (none) to 64.00% (structured), a smaller but still positive gain, which the paper reads as evidence that the effect is real but not uniform across models.

phoneshell's scaffold tunes efficiency, not state, and says so

phoneshell's (github.com/Blomega/phoneshell) own phoneshell/bench/runner.py defines a constant, SCAFFOLD_PROMPT, with a comment explaining why it exists at all: "A benchmark measures a model INSIDE a harness, and the harness always shapes the result, so this is stated rather than hidden: it describes what the tools are for and nothing about any specific task. Runs are labelled with whether it was used, and both numbers are published." The prompt's actual content, read in full from the source, tells the model to batch several tool calls into one phone_do round trip rather than observing between each tap, to stop observing "to confirm a step you have already been told succeeded," to use phone_set_picker instead of swiping at a spinning wheel because a swipe "steps by whole rows" and overshoots, and to call phone_dismiss_popup rather than hunting for a close button. None of those four instructions touches either of GMA's two measured levers. The scaffold does not tell the model how many past screenshots to retain, that choice is a property of the agent harness calling phoneshell, not of the scaffold text itself, and it does not give the model a structured field to write down current app state; it is pure efficiency guidance about which tool to call and when to stop observing.

The two projects are not measuring the same thing and neither invalidates the other. GMA isolates state-tracking and context retention specifically because its authors believe those are where harness design pays off most on complex, multi-step, cross-app workflows, exactly the tier (Realistic) where every one of GMA's eight frontier models scores worst, several at or near 0%. phoneshell's scaffold is tuned against a different cost: round trips and wasted observation calls against a real device, where every extra tool call is seconds of wall-clock time on actual hardware, not a free simulated step. A structured state field of the kind GMA tests would be a legitimate, separate addition to phoneshell's scaffold; it is not currently part of it, and the published scaffold/no-scaffold split in phoneshell's own results describes tool-use efficiency, not the state-tracking lever GMA isolates.

What it means for anyone tuning an agent's harness prompt

• GMA's result argues for measuring harness changes in isolation, the same model, varying one setting at a time, because the two levers it tested move the score by five to nine points each even though the underlying model never changes. A single pass/fail number without that breakdown conflates model capability with prompt engineering.
• The saturation in GMA's screenshot-count table (52.00% at both 8 and 16 images) is itself useful: past some point, retaining more context stops helping Qwen3.7-Plus at all. A harness designer chasing a bigger context window past that point is spending tokens for nothing on this specific lever and this specific model.
• phoneshell's publish-both-numbers policy for scaffold on/off is the right instinct GMA's own methodology validates, but the two numbers only tell a reader about the specific instructions in SCAFFOLD_PROMPT. They say nothing about whether adding explicit state tracking, the lever GMA found gave the largest single gain on Qwen3.7-Plus, would move phoneshell's own leaderboard.

Check it yourself

# phoneshell's own scaffold prompt, in full, and how it is labelled
grep -n "SCAFFOLD_PROMPT" -A 25 phoneshell/bench/runner.py
grep -n "scaffold" phoneshell/bench/runner.py

# confirm the scaffold never mentions app state or screenshot retention
grep -i "state\|screenshot\|retain" phoneshell/bench/runner.py

# GMA's own ablation tables
curl -s https://arxiv.org/html/2608.27477 | grep -i "state-tracking\|retained\|structured"
# full tables: https://arxiv.org/html/2608.27477 (Table 2, Table 3)

What would prove this wrong

We claim phoneshell's SCAFFOLD_PROMPT gives the model no structured place to track app state and no instruction about how many screenshots to retain. That is wrong the moment a future revision of phoneshell/bench/runner.py adds either to the prompt text. We also report GMA's structured-state gain on Qwen3.7-Plus as 5.00 points (49.33% to 54.33%, Table 3) and its screenshot-count saturation at 52.00% from 8 images onward (Table 2); both are wrong if a later revision of the paper restates those tables with different figures. We will check both by 31 March 2027.

Sources

• Zhu, Gao, et al., Benchmarking General Mobile Assistants in Challenging Real-World Scenarios, arXiv:2608.27477, submitted 21 August 2026, read 17 October 2026.
• Full tables: arxiv.org/html/2608.27477 (Table 1 per-tier scores, Table 2 context-retention ablation, Table 3 state-tracking ablation).
• Blolabel, phoneshell: phoneshell/bench/runner.py, read 17 October 2026.