How much of an AI agent's benchmark score is run-to-run noise?

In Wiedmann et al. (arXiv:2610.01618, October 2026), about 54% of score variance among runs that make a genuine attempt comes from repeating the same configuration, measured over 432 configurations run 5 times on each of four scientific tasks. Including all completed runs, configurations explain only 39.4% of variance.

Which configuration choice matters most for an agent?

In that study, the information given to the agent: it ranked first on every task, with effect sizes of 1.925, 1.452 and 1.496 within-cell standard deviations, against 0.704, 0.170 and 1.253 for model size. Prompting the agent to verify ranked last.

Did phoneshell's iPhone results change when rerun?

Of 144 task-model pairs run on 8 and 9 September 2026, 132 kept their outcome and 12 flipped, 11 from fail to pass. Fourteen commits landed in between, including lock survival and changes to the grading code, and 5 of the 12 flips involved the phone being found on the home screen.

How many runs per task does a phone agent benchmark need?

More than one. With single runs, noise, framework changes and device state cannot be separated from model differences. The paper uses 5 per cell and still reports wide confidence intervals.

← Notes

analysis 04 October 2026 8 min read

A new paper puts 54% of agent outcome variance in repetition; our next-day rerun of 144 iPhone runs flipped 12, and 11 flipped the same way

Agents Are Systems, Not Models measures how much of an agent's score is just rolling again, and the answer is over half. We checked the one place our public iPhone data repeats itself. The flips are real but lopsided, and they trace to framework commits and a phone that kept leaving the app, an axis the paper's sandbox holds fixed.

A paper posted on 1 October 2026 repeats every one of 432 agent configurations 5 times and finds that about 54% of score variance among genuine attempts is run-to-run noise, not configuration. phoneshell's public iPhone data has one repeat in it: 144 task-model pairs, run on 8 September and again on 9 September 2026. Twelve outcomes flipped, and 11 of them went from fail to pass, a split that symmetric noise produces with p = 0.0063.

Abstract technical illustration of two rows of small square cells, most matching between rows, a few changed in the same direction, over a faint grid

What Agents Are Systems, Not Models measured

Luis Wiedmann, Leander Girrbach, Cordelia Schmid and Zeynep Akata (TU Munich, Helmholtz Munich, Inria) give a coding agent four scientific tasks that each need a published specialist model: AstroCLIP for redshift, AstroSage-8B for MMLU-Astronomy (where the right move is to decline it), DNABERT-2 for promoter prediction and RiNALMo for RNA folding. They vary five axes over a full grid, 4 x 3 x 4 x 3 x 3 = 432 cells, run each cell 5 times, and get 2,160 runs per task and 8,640 in the main experiments. They release 18,240 trajectories in total.

Axis	Levels	Effect, promoter	Effect, redshift	Effect, RNA	Mean rank	Source
Information	none, identity, interface, protocol	1.925	1.452	1.496	1.00	Table 3
Model	Qwen3.5 35B-A3B, 122B-A10B, 397B-A17B	0.704	0.170	1.253	2.67	Table 3
Reasoning	act-only, think-act, ReAct	0.274	0.172	0.302	3.00	Table 3
Budget	5, 10, 20 min	0.142	0.201	0.255	3.33	Table 3
Verification	none, asked, reported, binding	0.131	0.130	0.077	5.00	Table 3
Between-cell share of variance: 39.4% all completed runs, 46.1% hurdle-clearing runs. Outcome consistency 0.86 pooled, 0.73 redshift						Fig. 2, Table 2

Effects are the range of mean gap closed across an axis's levels, divided by the within-cell standard deviation. Information ranks first in at least 99% of bootstrap samples. Asking the agent to verify barely changes whether it verifies; giving it a verification tool does. The finding that transfers to phones is the first one: a single run per configuration is mostly a draw.

The one place our data repeats itself

phoneshell's published results on Hugging Face (blolabel/phoneshell-bench) hold a 24-task sweep across six models in results/xv.jsonl, run 14:05 to 16:59 UTC on 8 September, and a 59-task sweep in results/xv2.jsonl, run 05:06 to 12:28 UTC on 9 September. Every one of the 144 pairs in the first appears in the second. That overlap is the only repeated configuration in the public data: the 349 rows of xv2 are one run per pair, and xv-text.jsonl is a copy of 48 xv rows with identical timestamps, not a rerun.

Paired, the 144 give 118 passes both times, 14 failures both times, 11 fail-to-pass and 1 pass-to-fail. If the flips were noise, each direction would be a coin toss; 11 against 1 has a two-sided sign-test p of 0.0063. Something systematic changed overnight, and the git log says what: 14 commits between the runs, including fd1798c "Store the passcode, survive a lock", f553248, which makes a long run heal itself (clear the lock screen, restore the port forward, restart the runner), and edits to bench/openrouter.py and bench/verify.py, the grading code.

The failure details agree. Five of the 12 flipped rows had a check reading "foreground is com.apple.springboard": the phone was on the home screen when the agent should have been in Calculator, Settings, Notes or Safari. That includes the single pass-to-fail row, Claude Opus 4.8 on notes.type_long, whose own report says it saved the note. A phone that leaves the app mid-task is a lock or a lost runner, which is what the overnight commits addressed. One more flip was the 240 s budget, one ran out of turns, and five were content checks (Calculator results, timer text, a Settings row, a Safari page).

Putting a number on it, with the caveats in the number

For pass-fail outcomes, a flip between two independent runs of the same cell happens with probability 2p(1-p), so half the flip rate estimates within-cell variance. Total variance over the 288 outcomes, at a pass rate of 248/288 = 86.1%, is 0.120. If all 12 flips were noise, within-cell variance is 12/288 = 0.042 and repetition is 34.8% of the total. If noise is symmetric and the one pass-to-fail flip is its whole share, there are about 2 noise flips and the share is 5.8%. The real figure sits between those, and the 11-to-1 direction argues for the low end. The paper's figure is 53.9% (100 minus 46.1) for genuine attempts. Ours is not comparable in kind: an 86% pass rate on short phone tasks leaves little variance to split, and the paper's tasks score in the middle where variance peaks (its Fig. 2 shows a Spearman rho of -0.79 between a cell's mean and its spread).

What this means for anyone ranking phone agents

The paper treats the sandbox as fixed and asks which knobs move the score. On a physical phone the sandbox is the least fixed part. Framework code and device state change between runs, and they change outcomes in one direction at a time, which noise does not. A leaderboard built from one run per cell folds all three sources into each number. Our own leaderboard note on 10 September found that no pair of the six models separated across 58 tasks; this rerun adds that part of what did move was the rig.

Run each cell at least twice, interleaved by model, so a framework or device change lands on every model equally instead of on whoever ran last.
Freeze the commit for a sweep and record it in every result row. xv2 rows carry device_model and device_os but no commit hash.
Report a "phone left the app" check separately from content checks. It identified 5 of 12 flips here and is the rig speaking.
If the paper's ranking carries over, the task prompt (what the agent is told) deserves more experiment budget than the model choice. phoneshell has not tested that.

Check it yourself

Download results/xv.jsonl and results/xv2.jsonl from https://huggingface.co/datasets/blolabel/phoneshell-bench, then: python3 -c "import json,collections;a={(r['task_id'],r['model']):r['passed'] for r in map(json.loads,open('xv.jsonl'))};b={(r['task_id'],r['model']):r['passed'] for r in map(json.loads,open('xv2.jsonl'))};print(collections.Counter((a[k],b[k]) for k in a))". Expect 118, 14, 11 and 1.
Sign test: python3 -c "from math import comb;print(2*sum(comb(12,i) for i in range(2))/2**12)" prints 0.00634.
The commits between runs: git log --since=2026-09-08T16:59:00Z --until=2026-09-09T05:06:00Z --oneline in the phoneshell repo (14 lines).
The paper's numbers: Table 2, Table 3 and Fig. 2 of https://arxiv.org/abs/2610.01618; trajectories at https://github.com/lusxvr/rethinking-agent-evaluation.

What would prove this wrong

If phoneshell reruns the 144 pairs twice more on one frozen commit with auto-lock off, and the flips between those two runs are again lopsided (one direction 10 or more out of 12 or fewer), then the overnight commits did not cause the 9 September flips and something else on the device drifts by day. We expect a frozen-commit repeat to show fewer than 6 flips, split roughly evenly. We will publish that repeat, or say we did not run it, by 31 December 2026.