GOLD beats SHAM by up to 25.9 points on a provenance audit; phoneshell's one answer-reading check was built to dodge exactly that confound

A paper posted 27 July 2026 names a confound most agent benchmarks never check for: a correct answer can mean the agent read the right value, or it can mean the agent already had a value and the environment happened to agree. Swapping in a matched wrong value instead of the right one, while changing nothing else about the task, moves success by 19.1 to 25.9 percentage points in the authors' own test. phoneshell's single check that ever grades what a model said, rather than what the phone shows, was restricted on purpose to the one case where this confound cannot arise: a ground truth the task itself fixes on a page it hosts.

Jingkun Luo and Da-Tian Peng's "Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation" (arXiv:2607.24054, submitted 27 July 2026) argues that outcome correctness is not the same claim as "the agent's reasoning used the information it was supposed to use." An agent can produce the right final answer by actually finding and reading the target value, or by already carrying a plausible guess that happens to match, or by exploiting some other artifact of the evaluation surface that has nothing to do with the task's intended information flow. Ordinary benchmarks cannot tell these apart because they only ever present the correct value. The paper's fix is a three-way substitution: a CLEAN condition that keeps only what the benchmark itself authorizes, a GOLD condition that makes the true target value available, and a SHAM condition that keeps the identical source structure and the identical exposure opportunity but swaps in a matched, incorrect value. If an agent's success rate depends on whether the value it was exposed to was actually correct, GOLD must outperform SHAM. If it does not, outcome correctness was never tracking what the benchmark assumed it was tracking.

What success provenance measures, and what D0 found

The authors call the thing a correctness-only score fails to capture success provenance: whether success depended on an acquired target at all, as opposed to following from something the agent already had or from a leak in how the surface is constructed. Their first experiment, D0, runs this GOLD-versus-SHAM substitution across four standardized evaluation surfaces and reports that GOLD exceeds SHAM by 19.1 to 25.9 percentage points. That range is itself the headline: on a benchmark where this test has never been run, a reported success rate could, in principle, already be sitting on contamination from either end of that gap, and nothing in a plain pass/fail score would show it. The paper's recommendation is not a new scoring formula bolted onto existing benchmarks; it asks that agent evaluations report success alongside whether the evaluated information state actually supported it, so a reader can separate "the agent used the right information" from "the agent happened to land on the right string."

Source: Jingkun Luo and Da-Tian Peng, "Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation," arXiv:2607.24054, submitted 27 July 2026, read 31 October 2026.

Why phoneshell's answer-reading check cannot have this problem

phoneshell's house rule, stated in bench/verify.py's own docstring, is that every check reads the phone directly rather than believing the agent's account of itself: "an agent that says 'I have enabled Do Not Disturb' and an agent that enabled Do Not Disturb are different things, and only the second one passes." There is exactly one exception, the answer_contains check, and its own comment names the reason it exists and the limit placed on it in the same breath: it is "the ONLY check that grades what the model said rather than what the phone shows, and it exists for one narrow case: a task whose deliverable is information, not a state change." The comment continues: "ground truth has to be fixed and known in advance, which is why these tasks run against a page we host rather than against content that can change."

That restriction is, without the paper's vocabulary, a working answer to the exact confound D0 measures. A GOLD/SHAM substitution test asks whether an agent's success tracks the true target or merely a plausible value it could have produced some other way. phoneshell's design removes the ambiguity at the source rather than auditing for it after the fact: the only tasks that ever use answer_contains point the agent at a page phoneshell itself hosts, with one value in it, so there is no live content that could change, no second plausible value the agent could already be carrying, and no SHAM condition to even construct. The check works because the environment that produces the "GOLD" value and the environment the task actually runs against are, by construction, the same four walls.

What this means for anyone writing an information-deliverable check

• Success provenance is a real gap in any benchmark that lets an agent see several plausible values before answering, including ones that let a model browse the web, read a document, or query a database it does not fully control. phoneshell's answer-reading tasks never give the agent that choice: the only value in play is the one the task hosts, so there is no second source for success to secretly depend on.
• This is a design decision with a cost, not a free lunch. By restricting answer_contains to pages it hosts itself, phoneshell gets a check immune to this confound at the price of never grading an agent's performance against the live, changing web, which is most of what a phone is actually used for day to day.
• The paper's broader point, that benchmarks should report success alongside whether the evaluated information state supported it, has no open item for phoneshell's answer_contains tasks specifically, because the information state there has exactly one possible value by construction. It would matter immediately if phoneshell ever added an information-deliverable task against live, uncontrolled content, which as of this writing it has not.

Check it yourself

# the one check that grades what the model said, and why it is restricted this way
grep -n "answer_contains" -A 14 phoneshell/bench/verify.py

# every other check kind dispatches against the live phone instead
sed -n '47,150p' phoneshell/bench/verify.py

# the paper itself
open "https://arxiv.org/abs/2607.24054"

# the D0 result as quoted in this piece
# "GOLD exceeds SHAM by 19.1 to 25.9 percentage points"

What would prove this wrong

We claim the paper's D0 experiment reports a GOLD-minus-SHAM gap of 19.1 to 25.9 percentage points across four standardized surfaces; that is checkable against the arXiv page itself and would be false if a later revision changes that range. We also claim phoneshell's answer_contains check is used only against pages phoneshell hosts with a single fixed value, which is checkable by reading every task file in environments/ that uses that check kind. This piece will need revisiting if phoneshell ever adds an answer_contains task against live, agent-browsable content, which would reopen exactly the confound this design currently avoids, or if the paper's D0 range is revised in a later arXiv version, by 30 April 2027.