MobileBench-OL injects noise at a 20% chance per step and costs its best agent 4.2 points; phoneshell's day-apart rerun flipped 12 of 144 outcomes with no noise injected at all

MobileBench-OL manufactures its own interruptions on purpose, at a known rate, so it can name exactly what cost an agent a point. phoneshell's own instability showed up uninvited, in a rerun nobody perturbed, and the harness has no equivalent name for what caused any of its 12 flips.

MobileBench-OL ("A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment," arXiv:2601.20335), submitted 28 January 2026 and revised the next day, scores 12 GUI agents across five subsets built from 1,080 tasks spanning 80 Chinese apps: Base, Long-Tail, Long-Horizon, GUI-Reasoning, and a fifth subset built around injected noise. That subset takes Base-subset tasks as seeds and, at every step of trajectory generation, injects one of four interruption types with a 20% chance: Repeat (the action fires more than once), Unexecuted (the agent's action is generated but never carried out), Delay (the page stays in a prolonged loading state), or Pop-up ("an unexpected dialog interrupts the workflow, e.g., a permission dialog or an ad"). The paper's best agent on the leaderboard, UI-TARS-1.5-7B, scores a 60.97% success rate on the clean Base subset and 56.77% on the noise-injection subset, a drop of 4.20 percentage points. Its Long-Horizon subset falls far further, to 15.00%, from the same 60.97% baseline, though that is a different stress (task length, not injected interruption) and not the subject here.

What changed, and when

MobileBench-OL's contribution is not just another leaderboard; it is a method for making instability reproducible. Its noise is injected deliberately, at a stated 20% per-step probability, drawn from four named categories, so a 4.20-point drop can be attributed to a specific, repeatable intervention rather than to whatever happened to go wrong on a given run. The paper pairs this with a reset mechanism: between task attempts, "a reliable agent" executes fine-grained inverse tasks to restore device state, catalogued as task-level resets (adjusting app state or user data for one task), app-level shared resets (such as "Clear Shopping Cart"), cases needing no reset, and cases marked infeasible.

phoneshell has no equivalent deliberate noise injection anywhere in its 80 tasks or its runner. What it does have is an uninvited natural-variance result: a day-apart rerun of 144 identical task attempts, reported elsewhere on this site, flipped 12 outcomes, 11 of them in the same direction, a pattern traced to the harness rather than to the model being tested. That rerun was never decomposed by cause. Nothing in phoneshell's logs says whether a given flip came from a pop-up, a delayed screen transition, a duplicated tap, or something else entirely, because the harness was not instrumented to tell the four apart, and no one perturbed it on purpose to find out which ones it is even sensitive to.

Named noise against unnamed noise

What it means for someone building or reading about these agents

MobileBench-OL's contribution here is the decomposition, not the headline drop. A 4.20-point fall from noise is a modest number by itself, smaller than many step-budget or vision-channel effects reported elsewhere on this site, but it comes with an answer to "which noise": Repeat, Unexecuted, Delay, or Pop-up, each with a known injection rate. That is a different, and in some ways more useful, kind of result than a larger but unexplained drop, because it tells an engineer which failure mode to harden against first.

phoneshell's 12-of-144 flip rate has never had that treatment. The result as published says the cause is "the harness," which is already more honest than blaming the model, but it stops one level short of naming which harness behavior: a popup the task did not expect, a load that took longer on the second day, a duplicated tap from a retry, or something the current instrumentation cannot see at all. Until phoneshell's own rerun logs are categorized the way MobileBench-OL's injected noise is, by type and by step, "the harness did it" is phoneshell's version of MobileBench-OL's four named categories collapsed into one, and that collapse is a measurable gap in what the harness currently reports about its own instability.

• A benchmark that injects noise on purpose can quantify exactly how much of its own instability comes from a specific, named cause. A benchmark that only observes natural variance, as phoneshell currently does, can report that instability exists without being able to say how much of it any single intervention would fix.
• MobileBench-OL's four noise types, repeat, unexecuted, delay, and pop-up, are a ready-made checklist for auditing any rerun discrepancy on another harness. phoneshell's own 12 flips have never been checked against that specific list.
• A deliberate, catalogued reset (MobileBench-OL's inverse-task design) is a different kind of asset than a reset discovered as a side effect of a bug (phoneshell's session-creation finding). The first is built to be trusted; the second was found because it was not.

Check it yourself

# MobileBench-OL's Table 5, per-subset scores for all 12 agents
curl -s https://arxiv.org/html/2601.20335v2 | grep -iE "60\.97|56\.77"

git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "Found in the first three runs of 120" FINDINGS.md   # phoneshell's own natural-noise catches
grep -n "## 6. Session management" -A5 FINDINGS.md           # the accidental reset
# phoneshell's day-apart rerun result (12 of 144 flips) is reported in this site's
# own VenusBench-Mobile comparison article; it has no per-cause breakdown to grep for,
# which is exactly the gap this article is pointing at.

What would prove this wrong

• We predict that if phoneshell's rerun logs were re-examined against MobileBench-OL's four noise categories, pop-up interruptions (confirm sheets, permission dialogs, promo overlays, already catalogued in phoneshell/agent/playbook.py) would account for more of the 12 flips than delay or repeat, because phoneshell's own overlay catalogue is larger and better understood than its handling of slow loads or duplicated taps. This is untested as of this writing and falsifiable by whoever runs the categorization, through 31 December 2027.
• We claim phoneshell has never deliberately injected noise to measure its effect. That is wrong the moment a future experiment adds a scripted delay, duplicated tap, or forced pop-up to a subset of its 80 tasks and reports a before-and-after score, the same design MobileBench-OL already ran.