VenusBench-Mobile: the best mobile agent passes 15% of its five-repeat stability tasks; phoneshell's own day-apart rerun flipped 12 of 144, 11 of them one way

Two different repeat experiments, two different kinds of instability, and neither one is what a single success-rate number reports. VenusBench-Mobile varies the task and demands five-for-five. phoneshell reran the identical task a day later and the flips traced to the rig, not the model.

VenusBench-Mobile, accepted at ICML 2026 (PMLR volume 306), scores its best agent pairing, Gemini-3-Pro paired with the UI-Venus-72B grounding model, at 36.9% total success across 149 single-run online Android tasks spanning 27 apps. On the benchmark's stability subset, which adds five variants of a subset of tasks and counts a pass only if the agent succeeds on all five, the same pairing drops to 15%, and most of the other eight tested agents score 0%. phoneshell's own repeat test is a different experiment answering a different question: 144 identical iPhone task runs executed a day apart, no intentional variation in the task or the starting screen, and 12 outcomes flipped between the two days. 11 of those 12 flips moved the same direction, and the article that reported them traced the cause to two commits in phoneshell's own framework and a phone that kept leaving the foregrounded app, not to anything resembling model randomness. Both results are real. Neither is "the" instability number for mobile agents, because they are measuring different sources of it.

What changed, and when

VenusBench-Mobile (Gong, Cai, Dai, Zhou, Gu, Meng, Shen) was submitted to arXiv in February 2026 (arXiv:2604.06182) and appears in the ICML 2026 proceedings via PMLR volume 306. It argues that prior online mobile benchmarks concentrate on single apps and repetitive task types, and builds a user-intent-driven task set with a capability-oriented annotation scheme meant to attribute each failure to a specific agent component rather than a bare pass or fail. Its diagnostics attribute most failures to perception and memory. Code and data are published on the UI-Venus GitHub repository's VenusBench-Mobile branch.

phoneshell's rerun was prompted by a separate paper, Wiedmann, Girrbach, Schmid and Akata, "Agents Are Systems, Not Models" (arXiv:2610.01618, 1 October 2026), which repeats 432 agent configurations five times each and attributes about 54% of score variance to run-to-run noise, with task identity outranking model identity as a predictor of that variance. phoneshell checked the one place its own public data repeats: 144 runs executed a day apart on the same physical iPhone, reported in the earlier note "A new paper puts 54% of agent outcome variance in repetition" (16 October 2026 at this outlet).

What each repeat test actually found

What this means for someone reporting a success rate

A pass rate reported from one run per task answers "did it work that one time," which is a narrower claim than most leaderboards present it as. VenusBench-Mobile's result says that even the strongest tested pairing is brittle to starting-state variation specifically: the same task phrased or begun five different ways succeeds all five times only 15% of the time, against 36.9% on any single attempt. phoneshell's result says a completely different thing: given the exact same environment and the exact same task, outcomes can still move between two calendar days for reasons that have nothing to do with the model, and the direction of that movement here was not random, it was a one-way correction once a bug got fixed. Averaged into one number, these would look like generic "noise." Reported separately, they point at different fixes: VenusBench- Mobile's number argues for training or prompting that generalizes across phrasing and starting state; phoneshell's argues for pinning framework versions and checking app foreground state before trusting any single run's score.

The Wiedmann et al. finding that about 54% of variance is run-to-run noise sits above both of these as a reminder that even after accounting for variation-brittleness and infra drift, a sizeable remainder is still unexplained repetition noise in the strict sense, the kind that would show up even with a perfectly pinned harness and an unvarying task. None of the three studies here claims to have isolated that remainder cleanly from the other two.

Check it yourself

git clone https://github.com/inclusionAI/UI-Venus -b VenusBench-Mobile   # VenusBench-Mobile's own tasks and data
git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -A10 "day apart" FINDINGS.md 2>/dev/null; grep -rn "144" FINDINGS.md
# phoneshell has no built-in task-variant generator as of this writing, so a
# VenusBench-style 5-variant rerun of its own 80 tasks does not exist as a one-line command

What would prove this wrong

• We predict that if phoneshell's 144 tasks were each run through five hand-written starting- state variants in the style of VenusBench-Mobile, the flip rate would exceed 12 of 144 (8.3%), because VenusBench-Mobile's stability drop (36.9% to 15%, a 59% relative fall for the best agent) suggests variation-brittleness is a larger effect on well-behaved tasks than the day-to-day infrastructure drift phoneshell happened to catch. This is untested, since the variant generator referenced above does not yet exist, and is falsifiable by whoever builds it, through 31 December 2027.
• If a rerun of phoneshell's same 144 tasks on a third day, with the two identified framework bugs now fixed, shows zero flips, that would confirm the 11-of-12 one-directional pattern was entirely the two named bugs and not a third, undetected source of drift. If flips persist at a similar rate with the known bugs fixed, that would mean something else is also moving outcomes day to day, and this article's attribution would need revision.