ComtradeBench's shipped harness owns all 5 execution-critical decisions, so 21 of 21 model pairs submit byte-identical answers; phoneshell's scaffold prompt makes none of a task's decisions for the model

A measurement-theory audit of agent benchmarks names a "double measurement confound": a fixed scaffold that makes a task's real decisions instead of the model, scored by a judge that grades shape instead of truth. On ComtradeBench's shipped configuration, the harness owns all five execution-critical decisions and 21 of 21 model pairs return byte-identical submissions. A fabricated 2,345-row answer scores 0.987 under the shipped judge, tied with the correct one. phoneshell's published scaffold prompt makes none of a task's decisions; the model owns every tap.

The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean (arXiv:2609.09218), submitted 6 September 2026 by Yonghong Zhang, Shadi Motaali, Vu Phong Dinh, Avin Piroutiniya, Jorge E. Lopez de Vergara, Luis de Pedro, Ricardo Correia, Isabel M. Parra, and Yong Xie, argues that an agent benchmark score is only evidence about the model if two conditions hold: the scaffold around the model is not making the task's real decisions for it, and the scorer is checking submitted values against ground truth rather than checking output shape. The paper tests both conditions directly on ComtradeBench, a benchmark built on seeded, synthetic trade-statistics data served by a mock API, by building an explicit decision-ownership map covering five execution-critical choices: retry, dedup, totals-drop, pagination, and submission. Under the benchmark's shipped, production harness (which the paper labels L0), the harness owns all five of those decisions. The model is never asked to make any of them.

Twenty-one model pairs, one answer, because the harness never asked them anything

The paper's starkest number is a null result dressed as data: when the production harness owns all five execution-critical decisions, 21 of 21 tested model pairs submit byte-identical outputs on the same task. Swapping in a different model changes nothing observable in the submission, because the model was never in a position to change anything; the harness had already decided what to retry, what to deduplicate, when to drop a totals row, how to paginate, and what exactly to submit. A second result shows the scoring side has an equally direct failure: on the task T2_multi_page, a submission of 2,345 fabricated rows, formatted to the correct schema, scores 0.987 under the benchmark's shipped judge, the same score the correct record set receives. Scored against ground truth instead, the fabrication gets an F1 of 0.000 and the correct set gets 1.000. The shipped judge, the paper notes, was grading "output format, field presence, or self-reported metadata," never the submitted values.

phoneshell's published scaffold prompt owns zero of a task's decisions, by design

phoneshell's scaffold prompt (github.com/Blomega/phoneshell, phoneshell/bench/runner.py) is published in full in its source, and its own docstring states the reasoning behind publishing it: "A benchmark measures a model INSIDE a harness, and the harness always shapes the result, so this is stated rather than hidden." Reading the prompt against the double measurement confound's five-decision checklist (retry, dedup, totals-drop, pagination, submission) finds no equivalent of any of them. The text tells the model how to batch tool calls efficiently, how to set a picker wheel, and to call a dedicated phone_dismiss_popup tool rather than hunting for a close button; it never decides, on the model's behalf, which element to tap, when a task is finished, or what answer to submit. The model owns every one of those calls. That is the condition the Double Measurement Confound paper's repaired ComtradeBench configuration had to be engineered to reach; it is phoneshell's starting position.

What it means for reading any agent leaderboard

• Two models tying, or nearly tying, on a benchmark is not evidence they perform alike. It can be evidence the harness decided the outcome before either model ran, which is exactly what a rule-based script with no language model scoring 96.8 against Kimi's and Claude's 97.5 demonstrates: the three are indistinguishable because none of them was doing the deciding.
• τ-bench's own numbers quantify how much of this risk is structural rather than ComtradeBench-specific: flipping only the scaffold flag shifts the same model's score by up to 0.267 and reorders the leaderboard, and a two-factor breakdown attributes more of the score's variance to the scaffold (η² = 0.0293) than to the model being tested (η² = 0.0080).
• The paper's "not probed is a first-class cell value" principle, that an unaudited item is never reported as passing, is the same discipline as phoneshell's rule that a task the model was never actually asked to attempt is reported as not scored, never as a failure or a pass. Both treat a missing measurement as a missing measurement, not a result.

Check it yourself

# the paper's decision-ownership map and ComtradeBench results
curl -s https://arxiv.org/abs/2609.09218
# full tables (E0/E1 leaderboards, tau-bench scaffold-flip, BFCL probe): https://arxiv.org/html/2609.09218

# phoneshell's scaffold prompt, published in full, and what it does NOT decide
sed -n '/SCAFFOLD_PROMPT/,/^"""/p' phoneshell/bench/runner.py

# confirm grading reads device state, never the harness's own report
grep -n "run_all\|CheckResult" phoneshell/bench/verify.py | head -10

What would prove this wrong

We report that ComtradeBench's shipped L0 harness owns all five of the paper's named execution-critical decisions and that this produces byte-identical submissions across 21 of 21 tested model pairs, both read directly from arXiv:2609.09218. That is wrong if a future revision restates the decision-ownership table differently, or if the 21 model pairs are later found to diverge once more of the paper's appendix is checked. We also claim phoneshell's scaffold prompt makes none of a task's decisions for the model, based on reading its full published text in phoneshell/bench/runner.py; that claim is wrong the moment a future commit adds any task-level decision, such as a fixed order of operations or a pre-chosen UI route, into that prompt. We will check both by 31 March 2027.

Sources

• Zhang, Motaali, Dinh, Piroutiniya, Lopez de Vergara, de Pedro, Correia, Parra, Xie, The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean, arXiv:2609.09218, submitted 6 September 2026, read 18 October 2026.
• Full tables: arxiv.org/html/2609.09218 (decision-ownership map, E0/E1 ComtradeBench leaderboards, τ-bench and BFCL audits).
• Blolabel, phoneshell: phoneshell/bench/runner.py, SCAFFOLD_PROMPT, read 18 October 2026.