A replay script that never looks at the screen beat the frontier model it copied on OSWorld and MobileWorld, then scored 6.9% against DigiWorld's 45.7%; phoneshell's setup step promises the exact start state a replay needs

"Computer Use at the Edge of the Statistical Precipice" (arXiv:2605.08261, Meta Superintelligence Labs, 7 May 2026) records one correct trajectory and replays it with no vision and no reasoning. It beats the frontier model it copied on two static benchmarks and falls to 6.9% on the one benchmark built to resist it. phoneshell's own task format guarantees the one property that makes the trick work, and has never tested for it.

"Computer Use at the Edge of the Statistical Precipice" (arXiv:2605.08261, submitted 7 May 2026 by Pierluca D'Oro and eight co-authors at Meta Superintelligence Labs) built a roughly 1MB script that looks at nothing: it replays a single recorded action sequence, open-loop, with no screenshot, no reasoning and no model call. On OSWorld it scores 71.1%, half a point above the 70.6% of the frontier model whose trajectory it copied. On MobileWorld it scores 41.5% against that model's 32.7%, 8.8 points ahead. On the authors' own DigiWorld, a benchmark built specifically to resist this, the same script collapses to 6.90% against a real model's 45.7%. phoneshell's own task format, by its own docstring's design, guarantees exactly the property that makes the first result possible and the second one necessary to test for.

What the paper shows, and why DigiWorld is different

The authors' argument is that a benchmark's expected score for an open-loop replay equals the source agent's own pass@k whenever the environment is deterministic, so a static benchmark cannot tell the difference between a model that reasoned its way to the answer and a script that memorized one correct path through an environment that never changes. DigiWorld is their answer: 15 hand-built Android apps across 7 domains, reconfigured into more than 3.2 million verified unique configurations, spanning 387 scenarios and over 4,300 task instances, so that no single recorded trajectory works twice. The paper proposes a second fix for a different problem, that benchmark rankings are usually reported as point estimates on too few runs to separate models that are not actually different: pairing Wilson score intervals with a hierarchical bootstrap that respects the nested structure of tasks inside apps inside configurations. In a worked simulation at 3 rollouts per task, the ordinary Wald interval covers the true value about 25% of the time; the Wilson interval the paper uses instead covers it about 95% of the time, and switching to the full hierarchical method is reported to cut a benchmark's total expected regret from 9.8% to 6.0% by refusing to declare a winner on three apps where only the weaker interval saw one.

Source: arXiv:2605.08261, Table 1, read 27 October 2026. Replay trajectories for OSWorld and DigiWorld came from Claude Opus 4.6; the MobileWorld replay trajectory came from Gemini 3 Pro.

Which side of this result phoneshell's own tasks sit on

bench/schema.py's own module docstring describes a phoneshell task's setup field as "deterministic steps that put the phone in a known start state, run with no model involved so every attempt starts identically." That sentence is a design decision made for a good reason, stated in the same docstring: a task's checks are what make it worth running automatically at scale, and checks only mean something if every run starts from the same place. It is also, by the Statistical Precipice paper's own logic, the exact property that let a recorded replay match or beat a reasoning model on OSWorld and on MobileWorld: an identical start state, run after run, is what makes one recorded path reliably work twice. Nothing in Task.load or Step.parse randomizes a task's starting configuration the way DigiWorld's 3.2 million configurations do. By this description alone, phoneshell's 80 tasks sit on OSWorld and MobileWorld's side of the result, the side a non-reasoning replay was able to match, not DigiWorld's side, the one built to resist it.

phoneshell's own statistics module, bench/stats.py, already does independently what the paper spends several pages arguing the field needs: its wilson() function computes a 95% Wilson interval rather than a point estimate, for the same reason the paper gives, that the normal approximation lies about the bottom end near a benchmark's typical high pass rates; its mcnemar_exact() function runs a paired, discordant-pairs test between two models on the same task set, the same structural move the paper's hierarchical bootstrap generalizes; and the module's own comment records the reason it exists, that an earlier version of this project published a leaderboard where six models sat on six percentages, three identical, with "not a single pair of those models separat[ing] at p<0.05." That half of the paper's prescription, phoneshell already follows, for reasons it arrived at the same way the paper did, by being burned by the alternative first. What phoneshell has never tested is whether an agent submission that is, in fact, just replaying a trajectory it already has, open-loop, would score anomalously well against its own deterministic setup.

What it means for a benchmark that grades real models honestly

• phoneshell's device-state checks are not fooled by a model that lies about what it did; verify.py's own docstring says as much. They have never been tested against a model, or script, that does exactly what it needs to do without reasoning about it at all, because it has seen this exact task before.
• The two phoneshell behaviours are not in tension. Grading from device state (phoneshell's existing defense against a model that misreports success) and resisting a memorized replay (the defense DigiWorld adds) are different problems; phoneshell has solid footing on the first and no footing at all on the second, because nothing about phoneshell's task set has ever needed to resist a submission that already has the answer.
• The held-out private task set that schema.py's load_all function keeps off this site exists for a related but distinct reason, to stop a model generation from memorizing the public answer key during training, not to stop a replay from winning on the public set once a correct trajectory for it has been recorded once, by any means.

Check it yourself

# phoneshell's own determinism guarantee
grep -n "deterministic steps" -A3 phoneshell/bench/schema.py

# phoneshell's existing Wilson interval and paired exact test
sed -n '1,40p' phoneshell/bench/stats.py
grep -n "def wilson\|def mcnemar_exact" phoneshell/bench/stats.py

# confirm no task-level randomization of starting configuration exists
grep -n "class Step\|class Task" -A10 phoneshell/bench/schema.py

# the paper's own replay-script result and DigiWorld's configuration count
curl -s https://arxiv.org/html/2605.08261 | grep -iE "3.2 million|replay"

What would prove this wrong

We claim phoneshell's deterministic setup step places its 80 tasks structurally alongside OSWorld and MobileWorld, the two benchmarks a no-reasoning replay matched or beat, rather than alongside DigiWorld, the one built to resist it, and that this specific vulnerability has never been tested on phoneshell's own task set. That is wrong if any phoneshell task already randomizes its starting configuration in a way we did not find, or if a run already exists, that we could not locate, testing an open-loop replay against phoneshell's current suite. It is proven wrong the useful way the day someone records one correct trajectory per phoneshell task and replays it open-loop against a freshly run setup: if it scores close to the real pass rate phoneshell currently reports for frontier models, the paper's result transfers; if it scores far lower, phoneshell's real device, with its own network timing, live notifications and animation jitter, is already doing some of DigiWorld's job by accident. We will treat that specific test, run by either side, as settling the question, by 30 April 2027.