PhoneWorld's hard tasks are not hard for frontier agents: 83.3% on its mock apps against 31.6% on AndroidWorld's hard tasks

Version 3 of PhoneWorld adds a 13-agent evaluation and a fidelity audit. Frontier agents score as well on its generator-labelled hard tasks as on its easy ones, and the rebuilt apps render 97.7% of real page visits but make only 73.0% of them work.

Seven general agents average 86.0% on PhoneWorld's easy tasks and 83.3% on its hard ones. On AndroidWorld the same seven average 86.0% easy and 31.6% hard. Version 3 of PhoneWorld (arXiv:2605.29486, posted 28 September 2026) reports both in one table: the mock apps' "hard" label barely lowers frontier-agent scores, while AndroidWorld's cuts them by 54 points. The same version's fidelity audit finds its rebuilt apps render 97.7% of observed pages but make only 73.0% of them work.

What version 3 added

PhoneWorld is from Yuxuan Liu and 23 co-authors (Renmin University of China, The Chinese University of Hong Kong Shenzhen, Wuhan University and others). It turns screenshots and trajectories recorded in real consumer apps into runnable, resettable Android mock apps: 34 apps across 16 domains, a 120-task audited benchmark (44 easy, 41 medium, 35 hard), a pool of 7,936 generated tasks, and 3,354 verified training trajectories. Each benchmark task is checked by a program: 49 compare an answer, 71 query the app's SQLite database.

Versions 1 (28 May 2026) and 2 (14 August) were about training data. Version 3 adds three things. There is an online evaluation of 13 agents on PhoneWorld and on AndroidWorld's 116 tasks. There is an RL experiment on real phones. And there is a four-app audit of how faithfully the mocks reproduce Douban, Xiaohongshu, Bilibili and Meituan Waimai. The version 2 HTML has none of these tables. The paper's comment line still reads "work in progress".

Rates are strict task success in percent: partial credit, execution errors and failed checks all count as failures.

The hard label means different things on the two benchmarks

Pooled, the seven general agents solved 204 of 245 hard PhoneWorld runs and 42 of 133 hard AndroidWorld runs. GPT-5.5 scores higher on PhoneWorld's hard tasks than on its easy ones (91.4 against 84.1), and so does Kimi-K3 (88.6 against 86.4). AndroidWorld's hard split is small, 19 task types, so one agent's 31.6% is 6 tasks. Even so, a 51.7-point gap in the mean across seven agents is far outside what 19 tasks can produce by chance.

The labels come from different places. In PhoneWorld, the task generator writes the goal, the step limit, the verifier and the difficulty in one pass (Appendix A.3; its example task, "search for Project Group and tap Favorite" in the QQ mock, is labelled easy with a 10-step limit). Our reading, which the paper does not state, is that a generator can only call a task hard in terms of what it built. The deeper, service-backed behaviour that makes a real task hard is exactly what the audit says the mocks abstract away.

Open-source GUI agents do slow down on PhoneWorld's hard tasks, from 56.3% to 37.7% on average, and PWAgent from 77.3% to 51.4%. So the label is not meaningless. It separates 7B models and does not separate frontier ones. The authors' own claim of a discriminative benchmark rests on the overall spread, 72.5% to 94.2% for general agents and 36.6% to 56.7% for open-source ones, which Table 1 does show.

What the mocks keep and what they drop

The fidelity audit uses 963 real trajectories, 6,442 page visits and 94 page types from the four apps, weighted by how often real users hit each page. Pooled, 97.7% of visits land on a page the mock renders, 92.4% on one that is reachable and usable, and 73.0% on one whose core action produces the expected state change. High-frequency transitions are recalled at 73.7%, recurring ones at 59.7%, and visual similarity is 65.0%. The weakest of the four apps reaches 51.0% functional coverage.

The audit and the difficulty result point the same way. Rendered pages are almost complete, working pages are three quarters complete, and the long tail of transitions is the least complete. A hard task in a real app usually lives in that tail. The authors describe the design as "selectively abstracting deeper service-backed functionality", which is a fair description of a training environment and a limit on a benchmark.

What it means if you train or rank phone agents

As training data, the paper's case is strong on its own numbers. At a fixed 36,193-step budget, swapping 10K AndroidWorld steps for PhoneWorld steps raised AndroidWorld success from 56.9% to 71.6%. Widening app coverage from 5 to 34 apps at a fixed 10K steps raised it from 61.2% to 71.6%. In RL, giving half of a 50-step budget to mock apps raised the real-phone average from 40.67% to 45.33% and AndroidWorld from 77.2% to 83.2%. Cross-app real-phone tasks did not improve, 20.0% to 18.0%.

As a ranking, use it with care. Rankings already flip between the two benchmarks: Claude Opus 4.8 leads Gemini 3.5 Flash on PhoneWorld (94.2% against 92.5%) and trails it on AndroidWorld (78.4% against 85.3%). The real-phone evaluation in Table 3 uses different model versions (Claude Opus 4.7, Gemini 3.1 Pro), so it cannot be paired with Table 1 row by row. It does show what real apps cost. On 150 human-judged real-phone tasks, no reference model beats 60% on any of the three subsets, and Claude Opus 4.7 scores 16.0% on cross-app tasks.

For our own benchmark the lesson is about labels. phoneshell's 80 tasks run on a physical iPhone against Apple's first-party apps and are graded from device state, but their difficulty labels are hand-set in each task's YAML: 15 easy, 58 medium, 7 hard, and medium is the schema default when a file says nothing. Our leaderboard prints the label beside how many of six models solved the task (phoneshell/bench/site.py), and nothing checks one against the other. The PhoneWorld result says to make that check: a label that frontier models ignore is describing the task text, not the task.

Check it yourself

• Paper: https://arxiv.org/abs/2605.29486v3; Table 1 (difficulty splits), Table 3 (real phone), Table 4 and Appendix C (fidelity), Appendix A.3 (task synthesis), Appendix B Table 5 (44 / 41 / 35 split).
• Version 2 for comparison: https://arxiv.org/html/2605.29486v2; its Table 1 is a suite summary with no agent scores.
• Means: average the seven general-agent rows of v3 Table 1 per column. Pooled hard counts: 35 x (0.857, 0.914, 0.886, 0.914, 0.829, 0.743, 0.686), rounded, sum to 204; 19 x (0.316, 0.632, 0.368, 0.316, 0.211, 0.211, 0.158) sum to 42.
• phoneshell labels: grep -h "^difficulty:" environments/*.yaml | sort | uniq -c prints 15 easy, 7 hard, 58 medium; grep -n "difficulty" phoneshell/bench/schema.py shows the medium default.
• Code and gated APKs: https://github.com/EthanLeo-LYX/PhoneWorld, https://huggingface.co/datasets/EthanLeoLYX/PhoneWorld-APKs.

What would prove this wrong

Dated 31 March 2027. If PhoneWorld's authors or anyone else relabel its 35 hard tasks by human judgement or by measured step count, and the seven general agents still score within 10 points of their easy-task mean on the relabelled hard set, then the flat curve is a property of frontier agents on consumer-app flows, not of generator labels. We expect the relabelled hard set to open a gap of at least 20 points. A future PhoneWorld version whose audit reports functional coverage above 90% on the same four apps, with the hard-task mean still above 80%, would also count against our reading.