iOSWorld ties 133 tasks to one invented identity, Jordan Avery, across 26 apps; its accessibility tree adds up to 26 points for frontier models and nothing for smaller ones

iOSWorld's hardest category is not the one with memory in its name. Across its best configuration, multi-app tasks score lower than memory and personalization tasks, 36.7% against 54.3%, and the accessibility tree that helps frontier models by up to 26 points does nothing measurable for smaller ones. phoneshell has no persona, no memory task, and no schema field for either.

iOSWorld, submitted to arXiv as 2606.09764 on 8 June 2026 by Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh and Ruslan Salakhutdinov, builds 26 custom iOS apps around one shared, fictional user identity: Jordan Avery. Contacts, payments, messages, travel bookings and social relationships are written consistently across those 26 apps the way they would exist on one real person's phone, which is what makes the paper's 133 tasks possible to split into three categories rather than one undifferentiated pile: 27 single-app tasks, 60 multi-app tasks spanning two to eight apps, and 46 memory and personalization tasks that ask an agent to find a pattern in Jordan Avery's own data without being told where to look for it.

What changed, and when

The paper's best-scoring configuration, Claude Opus 4.6 given both a screenshot and privileged XML accessibility-tree access, reaches 51.9% overall (rounded to 52% in the abstract). Broken down by category, that same configuration scores 81.5% on single-app tasks, 36.7% on multi-app tasks, and 54.3% on memory and personalization tasks. The ordering is not what the category names suggest: a task requiring an agent to infer a pattern across a person's own data, with no hint where to look, scores higher than a task that simply has to operate two to eight apps in sequence. Multi-app coordination, not memory, is where this benchmark's best model loses the most ground.

Where the accessibility tree helps, and the paper's own inconsistency about who counts as frontier

iOSWorld's introduction states that privileged vision+XML access, the accessibility tree layered on top of a screenshot, "improves the stronger frontier models by up to 26 percentage points, while smaller models do not show the same gain." That is a clean, falsifiable-sounding claim, and it is also one the paper does not apply consistently to the same set of models. Section 2 of the paper calls five proprietary models frontier, a set that includes GPT-5.4 Mini. Section 4.2 and the caption of its own Figure 8 instead group GPT-5.4 Mini with the smaller models. Whether GPT-5.4 Mini's score counts toward the "up to 26 points" frontier gain or the "no gain" smaller-model finding depends on which section of the same paper is read, and we could not resolve that from the sections available to this check.

What it means for anyone designing or reading a phone-agent benchmark

• Calling a task category "memory" does not predict that it will be the hardest one. iOSWorld's own numbers put multi-app coordination 17.6 points below memory and personalization for its best configuration; a benchmark's category names are a design choice about what the authors wanted to test, not a forecast of where models will actually struggle.
• A shared persona is what makes a memory task meaningful at all: without Jordan Avery's data being consistent across all 26 apps, there would be no cross-app pattern for a memory task to ask about in the first place. phoneshell's tasks, run against whichever real accounts happen to be logged into its one physical iPhone, have a real identity behind them, but nothing in the task schema treats that identity as a resource an agent is asked to reason over across apps, so phoneshell could not host an iOSWorld-style memory task without first adding a category the schema does not have.
• The frontier-versus-smaller-model distinction for accessibility-tree gains is exactly the kind of claim this site's house rule on uncertainty exists for: state it plainly when a paper's own sections disagree about which models the claim covers, rather than picking whichever reading makes the headline cleaner.
• phoneshell's single-app-per-task schema means it has never measured anything like iOSWorld's multi-app category, the one category where iOSWorld's best model actually does worst. Whatever phoneshell's own 80 tasks say about a model's reliability, they say nothing about whether that same model can carry state correctly from one app into a second one.

Check it yourself

# iOSWorld's own task breakdown and the frontier/smaller-model inconsistency
curl -s https://arxiv.org/abs/2606.09764 | grep -i "Jordan Avery\|133 tasks\|memory"
# full text with Table 3 and Figure 8: https://arxiv.org/html/2606.09764

git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "class TaskSpec\|app:\|difficulty" phoneshell/bench/schema.py   # one app field, no category, no persona
grep -rn "persona\|identity\|memory" phoneshell/bench/ || echo "no persona or memory concept in phoneshell's task schema"

What would prove this wrong

We claim phoneshell's task schema has no field representing a shared persona or a memory/personalization category. That is wrong the moment any field in phoneshell/bench/schema.py models cross-task identity or asks an agent to recall something from a different task; we found none as of 23 October 2026. We also report an unresolved inconsistency in iOSWorld's own text about whether GPT-5.4 Mini counts as a frontier or a smaller model for the accessibility-tree gain; that is wrong if a later revision of arXiv:2606.09764 reconciles sections 2 and 4.2, which we will check by 30 April 2027.