MobilePA-Bench splits every score into four weighted capabilities, memory worth 20% and sub-agent collaboration worth 10%; phoneshell's 80 tasks return one pass or fail, nothing underneath it

GPT-6-Astra's 76.87% on MobilePA-Bench is not one number measured once. It is a weighted sum of four separate capability scores, tool use carrying half the weight and sub-agent collaboration carrying a tenth, and nine of the fourteen models tested score below 70% on that composite. phoneshell's task schema has no equivalent field: every one of its 80 tasks collapses to a single pass or fail, with nothing recorded about which capability broke.

MobilePA-Bench, posted to arXiv as 2608.23035 by a team at Alibaba's MAI group and revised to its third version on 19 October 2026, runs a mobile planner agent through a stateful sandbox with live application databases rather than static screenshots, across 1,705 tasks spanning eight query-task domains and 212 mobile tools. Each task is matched to a fixed checker for its tool calls, its state changes and, where relevant, its interaction behavior. The paper's headline number, GPT-6-Astra's 76.87% overall score, is the output of a weighted composite: tool use counts for 50% of a task's score, memory usage and skill usage 20% each, and sub-agent collaboration the remaining 10%. phoneshell's own schema (phoneshell/bench/schema.py) defines a Task as an instruction, a setup, a checks list and nothing more granular; the runner that scores it produces one boolean, not four weighted ones.

What changed, and when

The paper's step budget is fixed and small: an episode ends when the planner emits a Finish action or reaches Tmax = 15 steps, a third of phoneshell's own default per-task budget of 25 (Task.max_steps in phoneshell/bench/schema.py) and a quarter of phoneshell's hard safety ceiling of 60 (SafetyConfig.max_steps in phoneshell/config.py). The paper states plainly that missing or invalid tool-call predictions count as failures, but it does not say, in the sections available to this check, whether an episode that silently exhausts Tmax without ever emitting Finish is scored as a failure outright or evaluated purely on whatever state it happened to land in. That is exactly the ambiguity phoneshell's own house rule exists to close: a task phoneshell never finishes because of a rate limit or a missing app is logged as not scored, never as a failure, and the distinction is enforced in code, not left to a sentence in a paper.

The composite phoneshell's schema has no field for

MobilePA-Bench's four-way split exists because a planner can reach the user-visible goal while skipping a step the task actually requires: completing a booking without the memory lookup that should have informed which option to pick, or firing a business tool before the route that was supposed to load first. Scoring tool use, memory, skill and sub-agent collaboration separately, then weighting them 50/20/20/10 into one overall number, is how the paper keeps a model from getting full credit for the right answer reached the wrong way. phoneshell's Task dataclass has no equivalent axis to split along: id, instruction, app, difficulty, a setup list and a checks list, reduced by the runner to one boolean. A model that fails a phoneshell task by forgetting something it saw three screens earlier and a model that fails it by tapping the wrong button produce the same record: passed=false, with whatever text happens to be in the individual check's note field, not a structured capability tag.

What it means for anyone scoring a mobile agent

• A single pass/fail number cannot say whether a model is bad at using tools, bad at remembering what it saw, or bad at the handoff to a sub-agent. MobilePA-Bench's weighting makes that diagnosis possible before it makes the headline number; phoneshell's schema would need a new field, not a new metric, to do the same.
• Weighting tool use at half the composite is a design choice, not a law of mobile agents, and it is one MobilePA-Bench's authors made rather than one a model's behavior dictated. A benchmark that weighted memory and sub-agent collaboration 50/50 instead would likely produce a different leaderboard order from the same fourteen models' same raw trajectories.
• The paper's silence on what happens when Tmax is hit without a Finish call is the kind of gap this site's own house rule treats as a real finding, not a rounding error: a budget-exhausted trajectory and a trajectory that confidently got the task wrong are different failure modes, and conflating them under one failure count makes the 9-of-14-below-70% figure harder to interpret than it looks.
• phoneshell's 80 tasks were built around one pass/fail check per task on purpose, to keep the checker itself auditable: a four-way weighted score multiplies the number of things that can be wrong in the grading code by four. That is a real tradeoff against MobilePA-Bench's finer-grained diagnosis, not a free win either way.

Check it yourself

# MobilePA-Bench's capability weights and step cap
curl -s https://arxiv.org/abs/2608.23035 | grep -i "tool use\|memory usage\|sub-agent\|Tmax"
# full text with the results table: https://arxiv.org/html/2608.23035

git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "class Task\|checks:\|max_steps" phoneshell/bench/schema.py   # one checks list, one max_steps field
grep -n "max_steps" phoneshell/config.py                              # separate 60-step hard ceiling
grep -rn "capabilit\|memory\|sub.agent" phoneshell/bench/ || echo "no capability field in phoneshell's task schema"

What would prove this wrong

We claim phoneshell's task schema has no field that scores a capability separately from overall pass/fail. That is wrong the moment any field in phoneshell/bench/schema.py or the runner that consumes it records a sub-score; we found none as of 24 October 2026. We also report that MobilePA-Bench's text does not specify how a Tmax-exhausted episode without a Finish call is scored; that is wrong if a later revision of arXiv:2608.23035 states it plainly, which we will check by 30 April 2027.