Claude Opus 4.8 lost 15 Terminal-Bench tasks to XYEval's bad advice and won back 10: its drop is inside the noise

We rebuilt per-task flip counts for XYEval's 20 pass-or-fail cells from two of its own tables. Every cell gains tasks under misleading advice as well as losing them, and the two drops no test resolves are the ones that make Opus look hardest to mislead.

Under XYEval's misleading suggestions, Claude Opus 4.8 lost 15 Terminal-Bench tasks it had solved and solved 10 it had failed. That nets to the 6.2-point drop in the paper's Table 1, and a paired McNemar test on 15 against 10 gives p = 0.42. We reconstructed those flip counts for every binary-scored cell of XYEval (arXiv:2609.23939, posted 20 September 2026) from its Table 1 and its solved-only Table 16. Every one of the 20 cells shows gains under bad advice as well as losses, which is what one run at temperature 1.0 looks like, and the paper has no rerun of the unmodified tasks to say how much of the churn is the advice.

Two papers in two days, one arithmetic

XYEval, by Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord and co-authors, takes six existing benchmarks and inserts a plausible but wrong suggestion from the user (the "XY problem": asking about your attempted fix rather than your problem). Table 1 gives the original score, the score under the misleading suggestion and the relative change for five models. The largest relative drop is 46.7%, Gemini 3.1 Pro on Terminal-Bench, 67.4 to 36.0. Tables 9 to 13 give the settings: temperature 1.0 for every model, and task counts that differ by model. Opus 4.8 and GPT 5.5 ran 200 SWE-bench Verified tasks against the Gemini models' 500, 500 HLE questions against 2,500, and 100 MCP-Atlas tasks against 500. The paper's text contains no variance estimate and no repeated run.

One day later, on 21 September 2026, Chetan Pathade, Prathamesh Pawar and Shubham Patil posted "Attack Success Rate Is Not a Number" (arXiv:2609.25173). Their meta-analysis of 259 agentic-security papers finds 65.3% report neither a variance estimate nor repeated runs for the headline metric, and only 29.7% of the papers that use an LLM judge report agreement with human labels. Their Section V gives the resolution of a single run over m units at 80% power and alpha 0.05: a difference no smaller than (1.96 + 0.84) × √(2p(1−p)/m). At a 30% base rate and m = 100 that is 18.2 points, and two systems 5 points apart are ranked backwards about 21% of the time. We recomputed both: 18.2 points and 22% under the same normal approximation.

What the paired counts show

Table 16 of XYEval reports the share of control-solved tasks each model fails under the suggestion. For a benchmark scored pass or fail, that share times the solved count is the number of tasks lost, and the difference from Table 1's XY score is the number gained. That works for SWE-bench Verified, SWE-bench Pro, Terminal-Bench and HLE. It does not work for τ²-bench, where the reconstruction gives negative gains (average reward is not a per-task 0 or 1), or for MCP-Atlas, which is scored by claim coverage. For those ten cells we report only the unpaired test. Counts are rounded from percentages, so a borderline p can move with a one-task rounding: Opus 4.8 on HLE ranges from 0.030 to 0.079 across ±1 on each count.

"Detectable gap" is the minimum difference an unpaired single run resolves at 80% power, from the formula above at the pair's mean score. Unpaired, six of 30 cells have p ≥ 0.05. Paired, two of those resolve: GPT 5.5 on SWE-bench Pro (22 lost, 3 gained, p = 0.0002) and Gemini 3.7 Flash on Terminal-Bench (15 lost, 3 gained, p = 0.0075). Opus 4.8 on HLE (57 lost, 37 gained) lands at p = 0.0495, which rounding can push either side of 0.05. Two stay unresolved: Opus 4.8 on Terminal-Bench (15 lost, 10 gained, p = 0.42) and Gemini 3.7 Flash on SWE-bench Pro (8 lost, 11 gained, p = 0.65, a net gain). Opus 4.8 on MCP-Atlas, 100 tasks and a 4.8-point drop, cannot be reconstructed and is unresolved unpaired (p = 0.41).

The gains column is the noise floor the paper did not measure

Nothing in a misleading suggestion should help an agent solve a task it failed without one. Yet every binary cell has gains, from 2 tasks (Gemini 3.1 Pro, Terminal-Bench) to 153 (Gemini 3.7 Flash, HLE). At temperature 1.0 a second run of the unmodified benchmark would also flip tasks in both directions, and without that run the losses in Table 16 mix advice-induced failures with ordinary run-to-run churn. Opus 4.8's solved-only drop on Terminal-Bench is 25.5%; its net relative drop is 9.1%. The gap between those two numbers is the churn, and the paper reports the first as the effect.

This bears on the paper's most quotable comparison. Opus 4.8 has the smallest relative drop in three rows: τ²-bench (11.2%), Terminal-Bench (9.1%) and MCP-Atlas (6.0%). Its τ²-bench drop, 10.0 points on 278 tasks, is solid. The other two are the two cells in the table that no test resolves, run on the paper's two smallest task sets, 89 and 100. "Opus resists bad advice best" rests on one benchmark, not three.

The curve is the unpaired resolution at three base rates, and it flattens slowly: halving the detectable gap takes four times the tasks. At a 50% base rate it is 39.6 points at 25 tasks, 19.8 at 100, 14.0 at 200, 8.9 at 500 and 6.3 at 1,000. Pairing does better whenever losses and gains are lopsided, as GPT 5.5 on SWE-bench Pro shows, and no better when they are balanced, as Opus on Terminal-Bench shows. McNemar only sees the flips.

What this means on an 80-task phone benchmark

Our public suite is 80 tasks on a physical iPhone, graded from device state. At a 50% base rate an unpaired run of 80 tasks resolves 22.1 points; at 90%, where frontier models sat on our last sweep (89.7% to 96.6%), it resolves 13.3. phoneshell/bench/stats.py runs McNemar's exact test instead, and min_discordant_for_significance() returns 6: no pair separates at p < 0.05 with fewer than six one-directional flips, whatever the suite size. On our 58-task sweep, 46 tasks were passed by every model and no pair separated.

XYEval's gains column names the step we have not taken either: a rerun of the same model on the same tasks with nothing changed. Without it, a McNemar test compares two conditions but cannot say how many flips the phone, the network and sampling produce on their own. Two decisions follow for anyone running agent ablations. Publish the per-task pass/fail matrix, because Table 16 was only useful here by accident of how it was defined. And run the control condition twice; the flips between those two runs are the floor any claimed effect has to clear.

Check it yourself

curl -sL https://arxiv.org/html/2609.23939v1 -o xy.html      # Table 1, Tables 9-13, Table 16
curl -sL https://arxiv.org/html/2609.25173v1 -o asr.html     # Section V formula, 18.2 pp at m=100
python3 - <<'PY'
import math
from math import comb
def mcnemar(b, c):
    n, k = b + c, min(b, c)
    return min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / 2**n)
def flips(orig, xy, solved_only_drop, n):
    solved = round(orig / 100 * n)
    lost = round(solved_only_drop / 100 * solved)
    gained = round(xy / 100 * n) - (solved - lost)
    return lost, gained, round(mcnemar(lost, gained), 4)
print(flips(67.9, 61.7, 25.5, 89))    # Opus 4.8, Terminal-Bench  -> (15, 10, 0.4244)
print(flips(57.5, 48.0, 19.1, 200))   # GPT 5.5, SWE-bench Pro    -> (22, 3, 0.0002)
print(flips(45.5, 41.6, 24.9, 500))   # Opus 4.8, HLE             -> (57, 37, 0.0495)
mdd = lambda p, m: 2.801585 * math.sqrt(2 * p * (1 - p) / m) * 100
print(round(mdd(0.3, 100), 1), round(mdd(0.5, 80), 1), round(mdd(0.9, 80), 1))  # 18.2 22.1 13.3
PY

# our paired floor, from the phoneshell repository root
python3 -c "from phoneshell.bench.stats import min_discordant_for_significance as m; print(m())"  # 6

What would prove this wrong

The claim is that Claude Opus 4.8's Terminal-Bench and MCP-Atlas drops, and Gemini 3.7 Flash's SWE-bench Pro change, are not resolved by the runs XYEval reports, and that its solved-only drops include run-to-run churn. It is wrong if the authors release per-task outcomes and the true flip counts give McNemar p < 0.05 for Opus on Terminal-Bench, or if a second control run of any model shows fewer than a quarter as many flips as its control-versus-XY comparison. The second would mean the gains column is mostly an effect of the advice, not noise. We will check the paper's repository and any v2 on 31 December 2026 and publish what we find, including a note that nothing was released if that is the case. On our side, the next sweep will include one repeated control run per model.