VenusBench-Mobile's best agent holds 15% of its tasks stable under a theme or language change, most hold 0%; phoneshell has run its 80 tasks only one way
VenusBench-Mobile retests each of its 149 base tasks four more ways: in Chinese, in a paraphrased English instruction, in dark theme, and on a tablet screen. Its Stability Pass Rate, the share of tasks an agent gets right in every one of those variants, reaches 15% for the best model tested and 0% for most. None of phoneshell's 80 environment files vary theme, language, or device shape at all.
VenusBench-Mobile, accepted as an ICML 2026 oral and posted to arXiv on 6 February 2026 by Yichen Gong, Zhuohan Cai, Sunhao Dai, Yuqi Zhou, Zhangxuan Gu, Changhua Meng and Shuheng Shen (Tsinghua University and Ant Group), scores two different things per task. One is whether an agent completes a task at all. The other, Stability Pass Rate (SPR), is whether it completes the SAME task again under four perturbations the paper adds on top of its 149-task primary pool: a Chinese translation of the instruction, a semantically equivalent English rewrite, a dark-theme render of the same screens, and the same app on a tablet at 2560x1600 in landscape. Twenty instances of each, drawn from the same base tasks, give a 100-instance subset (20 originals plus 80 variants) built specifically to measure whether a pass was luck or competence.
What SPR actually penalises, and what it does not
SPR is unforgiving by construction: an agent only earns credit for a base task if it also passes all four of that task's perturbed variants. Table 3 of the paper gives per-model total success rate on the full 149-task pool. The best single-model result is UI-Venus-72B at 15.4%. The best result of any kind is a planner-executor pairing, Gemini-3-Pro as planner with UI-Venus-72B as the grounding executor, at 36.9%, with GPT-5.1 paired the same way reaching 26.9%. Every other model tested, run alone, falls between 6.7% and 12.8%. The paper's own text states the perturbation result plainly: most models reach an SPR of 0%, and even the best-performing model reaches only 15%. Visual layout changes, the tablet condition specifically, produce the largest drop of the four perturbation types for every model tested. The paper's diagnosis is that failures are dominated by perception and memory deficiencies that a coarse, pass/fail-only evaluation would never surface, because a model that happens to pass the one version of a task it was shown looks identical, on that metric alone, to a model that would pass it under any phrasing.
| Model | Total success (149 tasks) | Setup | Source |
|---|---|---|---|
| Gemini-3-Pro + UI-Venus-72B | 36.9% | planner-executor pair | arXiv:2604.06182 Table 3 |
| GPT-5.1 + UI-Venus-72B | 26.9% | planner-executor pair | arXiv:2604.06182 Table 3 |
| UI-Venus-72B | 15.4% | single model | arXiv:2604.06182 Table 3 |
| MAI-UI-8B | 12.8% | single model | arXiv:2604.06182 Table 3 |
| Mobile-Agent-v3 (GUI-Owl-7B) | 12.1% | agentic framework | arXiv:2604.06182 Table 3 |
| UI-Venus-7B | 8.1% | single model | arXiv:2604.06182 Table 3 |
| Qwen3-VL-30B-A3B | 8.7% | single model | arXiv:2604.06182 Table 3 |
| Qwen3-VL-8B / GUI-Owl-7B | 6.7% | single model | arXiv:2604.06182 Table 3 |
| Best model, Stability Pass Rate | 15% | all 4 perturbations must hold | arXiv:2604.06182 §5 |
| Most models, Stability Pass Rate | 0% | all 4 perturbations must hold | arXiv:2604.06182 §5 |
phoneshell has never asked the question this benchmark asks
None of phoneshell's (github.com/Blomega/phoneshell) 80 environment files in environments/ reference device orientation, dark mode, appearance, locale, or a second device shape. A grep for those terms across every YAML task file returns zero matches. Every run reported on this site uses one physical iPhone 17 Pro Max, in whatever appearance and language it happened to be set to that day, and each task is attempted exactly once per model per run; results.json keeps a single pass/fail row per attempt, with no field for a second, varied attempt at the same instruction. A model that passes "Open the Calculator, then open Safari, then go back to the Calculator" (environments/multi_app_switch.yaml) is scored as having solved that task, full stop. Nothing in the harness asks whether it would still pass if the instruction were rephrased, the phone were in dark mode, or the task ran on an iPad instead.
This is not a flaw unique to phoneshell. iOSWorld (arXiv:2606.09764), PhoneWorld (arXiv:2605.29486) and MobileGUI-VBench (arXiv:2609.12394), the other benchmarks this site has covered in depth, score the same way: one instruction, one environment, one attempt, one pass/fail outcome. VenusBench-Mobile's contribution is showing what that design choice hides. A model's reported success rate is not just an estimate of how often it succeeds; on this evidence, it may be close to an estimate of how often it succeeds specifically on the exact phrasing and exact rendering it happened to be tested with. The planner-executor result makes this concrete: a system that plans and grounds separately, Gemini-3-Pro paired with UI-Venus-72B, beats single models by more than double on the base task set (36.9% against a next-best single-model score of 15.4%), but the paper does not report that pair's own SPR, so whether its advantage on the base task survives a dark-theme render or a Chinese translation is, as far as this source goes, not reported.
What it means for anyone grading a phone agent
- A single pass/fail number per task, which is what phoneshell, iOSWorld, and most mobile agent benchmarks report, cannot distinguish a model that understood the screen from one that memorised the exact phrasing and rendering it was shown. VenusBench-Mobile's SPR exists specifically to separate those two things, and on its evidence the gap between them is large: a 15.4% single-model success rate on the base task corresponds to an SPR of 15% or less for the best model and 0% for most.
- The single largest perturbation effect in the paper is the device-shape change, tablet versus phone, not the language change. For anyone building an iOS-specific benchmark, that is a reason to treat "the task passed on an iPhone" and "the task would pass on an iPad" as two separate claims, not one.
- Adding a stability check to an existing 80-task suite does not require new tasks, only repeated runs of the existing ones under a controlled variation: same instruction translated, same screens re-themed, same app on a different device. phoneshell has the infrastructure to repeat a task (the
setup/checks/teardownstructure in every environment YAML already supports multiple runs); it has simply never been pointed at the same task twice with one variable changed.
Check it yourself
# confirm no phoneshell task varies theme, locale, or device shape
grep -rliE "orientation|dark.?mode|locale|appearance|ipad|tablet" environments/*.yaml || echo "no matches"
# confirm every task is scored from a single run, not paired variants
grep -n "passed\|turns\|seconds\|checks" phoneshell/bench/schema.py | head -10
# VenusBench-Mobile's own numbers
curl -s https://arxiv.org/abs/2604.06182 | grep -i "stability\|perturbation"
# full results table (Table 3) and the SPR definition (section 5):
# https://arxiv.org/html/2604.06182What would prove this wrong
We claim phoneshell has never tested any of its 80 tasks under a language, theme, or device-shape variation. That is wrong the moment a future run adds a second rendering of an existing task's instruction or runs the suite on an iPad, and either is reflected in results.json. We also claim VenusBench-Mobile's Stability Pass Rate, not its base success rate, is the more informative number for judging whether a model generalises. That claim is falsified if a later version of VenusBench-Mobile, or an independent replication, finds a model whose SPR tracks its base success rate closely (within a few points) rather than collapsing toward it. We will treat either event, checked by 31 March 2027, as grounds to revisit this comparison.
Sources
- Gong, Cai, Dai, Zhou, Gu, Meng, Shen, VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics, arXiv:2604.06182, submitted 6 February 2026, accepted ICML 2026 (PMLR 306:35862-35887), read 16 October 2026.
- Full text, Table 3, and the Stability Pass Rate definition: arxiv.org/html/2604.06182.
- Blolabel, phoneshell:
environments/,phoneshell/bench/schema.py, read 16 October 2026.