PrefDiag-Bench's best model clears 47.17% of Process preferences even with the profile handed to it directly; phoneshell has zero of its 80 tasks built to need one

The agent was not asked to infer what the user wanted. It was handed a written profile and still failed more than half the time on one preference type. The bottleneck is not reasoning about preferences, it is finding the right fact and knowing when it applies. phoneshell has no task that creates either problem.

A benchmark posted four days ago, "Diagnosing Personalization Failures in Mobile GUI Agents" (PrefDiag-Bench, arXiv:2610.30573, Zhixin Lin, Dongliang Xu, Jungang Li, Shidong Pan, Jun Feng, Yanbin Sun, Jing Li, Liang Liu and Yue Yao, submitted 22 October 2026), runs 1,664 stage-wise probes across five preference types against 11 agent systems, under both an explicit user profile and a behavioral memory log the agent has to read preferences out of itself. The headline number is specific and uncomfortable: Seed-2.0-Pro, the best-performing model the paper reports, clears only 47.17% of Process-type preferences even when the profile is handed to it directly rather than inferred. Agents complete non-personalized tasks at comparatively high rates; the moment a task requires a preference to shape how a multi-step action gets carried out, performance drops hard, and Ranking and Process preferences are the two hardest categories.

What changed, and when

PrefDiag-Bench's design isolates the question that matters: is a personalization failure a reasoning failure, or a plumbing failure. The paper's own diagnosis names two bottlenecks, selecting the relevant evidence out of a user's history, and recognizing when a given preference should actually apply to the current step, and its own controlled intervention supports that reading over a reasoning-deficit story: supplying the relevant evidence directly, rather than making the agent retrieve it, raises end-to-end task success by up to 7.06 points. That is a retrieval-and-timing problem dressed up as a capability problem in every prior benchmark that only measured overall task success rather than diagnosing the stage at which execution actually broke.

phoneshell's 80 tasks were never built to have this problem, in either direction. Every task starts from a deterministic setup state, carries one sentence of instruction, and is graded by reading the device afterward; covered here on 24 October, phoneshell has no persona, no multi-app memory category and no cross-app identity in its schema at all, the same structural gap iOSWorld's invented identity Jordan Avery exposed from a different angle. PrefDiag-Bench adds a sharper version of that same gap: not just "no persona," but no preference field, no behavioral log, and no mechanism by which an instruction could even specify "the way I usually do this." Of the five preference types PrefDiag-Bench diagnoses, Content, Ranking, Process, Timing and whatever the fifth category contributes to its 1,664 probes, phoneshell's current task definitions have an equivalent for exactly zero.

What the benchmark measures against what phoneshell has to measure it with

What it means for someone building or scoring a personalized phone agent

The useful finding here is not "agents are bad at preferences," which was already assumed by anyone who has watched one. It is that the badness is disproportionately a plumbing problem: an agent with the preference written down in front of it, no inference required, still gets the hardest category wrong more than half the time, and a large chunk of the gap closes when someone else does the evidence retrieval for it. That reframes where engineering effort should go for anyone building a personalization layer onto a GUI agent. The expensive-looking problem, teaching a model to reason about what a user probably wants, is smaller than the cheap-looking problem of building reliable retrieval over a behavioral log and a trigger for when a stored preference is actually relevant to the step in front of the agent right now.

For phoneshell, the finding does not land as a failure, because the benchmark measures a capability phoneshell was never built to have. Every one of its 80 tasks is a closed, single-session, single-instruction unit, by design, which is exactly what makes its device-state grading reproducible: there is no history to misread and no stored preference to misapply, because there is no history or stored preference at all. That is a real limitation relative to what PrefDiag-Bench diagnoses, not a different, unrelated design choice. A phone agent that has to respect "the way I usually book a flight" needs a memory substrate and a retrieval step before the question PrefDiag-Bench asks becomes answerable one way or the other, and phoneshell's current schema has neither.

• "The model can see the preference and still gets it wrong" is a stronger result than "the model has to infer the preference and gets it wrong," because it isolates execution from perception. PrefDiag-Bench's explicit-profile condition is the cleaner of its two conditions for that reason, and 47.17% is the number from that cleaner condition, not the harder behavioral-log one.
• The 7.06-point gain from a retrieval intervention is a ceiling on how much of the problem is retrieval, not a floor; the paper does not report what fraction of the remaining failures after that intervention is a true reasoning failure versus a timing failure, so "fix retrieval and you are most of the way there" is not yet a licensed conclusion.
• phoneshell's gap here is the same shape as its memory and persona gap, cited above, but sharper: it is not merely missing a feature PrefDiag-Bench tests, it is missing the data structure, a behavioral log or stored profile, that any version of this feature would need to exist first.

Check it yourself

# PrefDiag-Bench's abstract and probe design
curl -s https://arxiv.org/abs/2610.30573

git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "persona\|profile\|preference\|memory" environments/*.yaml   # expect zero matches
cat README.md | sed -n '98,133p'   # the task schema: setup, instruction, checks, nothing else

What would prove this wrong

• We read 47.17% on Process preferences, with the profile supplied directly, as evidence that execution under a known preference is still substantially unsolved, independent of whether the preference had to be inferred. That is wrong if PrefDiag-Bench's full paper, once its tables are read past the abstract, shows a different model clearing Process preferences above roughly 65% under the explicit-profile condition; if so we will revise this article to say the ceiling is higher than the abstract's headline model suggests. Untested as of this writing, through 31 December 2027.
• We predict that the first personalization task phoneshell adds, whenever that happens, will fail primarily on evidence selection rather than on execution once the correct preference is known, mirroring PrefDiag-Bench's own diagnosis rather than phoneshell's usual failure shape of a stack that silently reports success. That prediction is falsifiable the day such a task exists and is run.