AgentHorizon builds failures by swapping instructions; 11 of phoneshell's 80 device checks pass a sibling task's finish

A judge that reads a trajectory can be fooled by a coherent finish to the wrong request. A grader that reads the phone can be too, and the swap test finds where without recording anything.

AgentHorizon, posted on 8 October 2026, tests judges of computer-use agents with a simple trick: record a task done right, then pair the recording with a closely related instruction it does not satisfy. The strongest judge, GPT-5.5 running inside OpenAI's Codex agent, reaches 80.9% balanced accuracy on the hard split. We ran the same swap over phoneshell's own 80 iPhone tasks, which are graded from device state rather than by a model. In 46 ordered task pairs, any finish that passes one task passes the other on check text alone. In 40 of those the instructions disagree. The 11 tasks involved passed 66 of 66 runs on 9 September 2026.

What AgentHorizon measured, and how the negatives were built

Xing Han Lù and eleven co-authors at ServiceNow Research, Mila and McGill (arXiv:2610.11050) collected 166 hours of human screen recordings on Windows, Linux and macOS. Annotators wrote two instructions that differ in one requirement and recorded a demonstration of each. Matched pairs are positive candidates. Crossed pairs, recording B judged against instruction A, are negatives by construction: "a very coherent execution, but it is paired with a request that the execution does not satisfy." After review the benchmark holds 1,373 items, 523 positive and 850 negative, with a median of 121 steps per item. Negatives are typed as Critical Mistake, Bad Side Effect or Misunderstanding.

Eleven judges were run directly and as coding agents inside five agent tools (Claude Code, Codex, Gemini CLI, OpenHands, OpenCode). Judges with similar balanced accuracy make opposite errors. Qwen 3.6 27B as a direct judge accepts 93.8% of valid work and rejects only 61.5% of swapped work. GPT-5.5 in Codex rejects 90.0% of swaps and accepts only 71.8% of valid runs. Given no instruction at all, Qwen 3.6 27B calls 98.9% of both classes successful, which is the paper's evidence that a finished-looking trajectory carries almost no signal by itself. Bad Side Effect had the lowest recall of the three types.

The same swap, applied to a grader that reads the phone

phoneshell's checks are assertions run against the phone after the agent stops: which app is in front, whether a text appears in the accessibility tree, whether a regular expression matches the screen, and for 20 probe tasks whether the answer contains a fixed word. Element matching is a case-insensitive substring over label, name, value and identifier (find_by_text in perception/tree.py). That makes implication between two tasks decidable from the YAML. Task A is implied by task B when every check in A is the same as a check in B, or, for element checks, is a substring of one. If A is implied by B, any phone state that passes B passes A. No recording is needed.

Over the 80 files the test finds 46 ordered pairs. We read each pair's two instructions. In 6, B's finish does satisfy A's instruction: a settings.about.serial run does reach the page settings.about.reach asks for. In the other 40, it does not. Those are AgentHorizon negatives that phoneshell's grader would score as passes.

Where the device stops being the witness

The 11 tasks are not failing. All 66 runs on 9 September passed, six models on an iPhone 14 Pro. Read the models' own summaries and they say the right things: all six settings.about.serial answers give MQ163LL/A, all six safari.read.heading answers say "Example Domain", every Spotlight run says it used Spotlight. The problem is where that evidence comes from. On these 11 tasks the part that separates the instruction from its sibling is confirmed only by the model's account, which is the source phoneshell exists to replace. On the same run, the models' summaries claimed success on 16 of 27 device-graded failures.

The three mechanisms are different. A path requirement (35 pairs) is invisible to any end-state check: Calculator in front looks the same with or without a trip through Safari. An answer requirement (3 pairs) is a missing check: phoneshell already has answer_contains and uses it on its 20 probe tasks, where the grader rejects swaps between the image word and the text word on the same page. A create-or-find pair (2) is a side effect: a model asked to search Notes can type a new note titled "phoneshell benchmark" and pass. Two of the six notes.search summaries say they found "multiple notes"; our guess, not checked on the phone, is that earlier create runs left them there.

Without the 11 tasks the 9 September run scores 256 of 283, 90.5%, against 322 of 349, 92.3%, with them. The 7 October note found 12 checks that pass on the opened app before any choice. The two lists measure different weaknesses, and we have not merged them.

What it means for anyone grading agents on phones

• Run the swap over your own task file before trusting a pass rate. For script graders it costs one loop over the checks and finds every pair the grader cannot separate. For LLM judges it costs one recording per sibling, which is what AgentHorizon paid for.
• A path requirement needs a path check. phoneshell's bridge already writes app launches, taps and typing to audit.jsonl, though without a run id yet (our 6 October note); grading "opened Safari between two Calculator states" or "typed into the Settings search field" from that log keeps the device as the witness. The model's summary should not be the evidence.
• Count what a check cannot reject, not only what it passes. AgentHorizon's split of Pos and Neg accuracy applies to script graders too. On these 40 pairs phoneshell's negative accuracy is 0 by construction.
• Side effects need their own checks. The notes pair is AgentHorizon's weakest category, Bad Side Effect, reproduced in a grader with no model in it. A check that the number of notes titled "phoneshell benchmark" did not change would catch it.

Check it yourself

# AgentHorizon: construction (Sec. 3.2), Table 1 counts, Table 2 judges
https://arxiv.org/abs/2610.11050
git clone https://github.com/ServiceNow/agenthorizon

# phoneshell: the implication test over the 80 task files (needs PyYAML)
python3 - <<'EOF'
import yaml, glob
T = {}
for f in glob.glob("environments/*.yaml"):
    d = yaml.safe_load(open(f))
    T[d["id"]] = [(c["kind"], str(c.get("text", c.get("value", ""))), bool(c.get("negate"))) for c in d["checks"]]
EL = ("element_exists", "element_exists_anywhere")
def implied(c, B):
    k, t, n = c
    for k2, t2, n2 in B:
        if n != n2: continue
        if n: 
            if k == k2 and t2.lower() in t.lower(): return True
            continue
        if (k, t) == (k2, t2): return True
        if k in EL and k2 in EL and t.lower() in t2.lower() and not (k == EL[0] and k2 == EL[1]): return True
        if k == k2 == "answer_contains" and t.lower() in t2.lower(): return True
    return False
P = [(a, b) for a in T for b in T if a != b and all(implied(c, T[b]) for c in T[a])]
print(len(T), len(P))          # 80 46
for a, b in sorted(P): print(a, "<=", b)
EOF

# the 9 September run
curl -sLO https://huggingface.co/datasets/blolabel/phoneshell-bench/resolve/main/results/xv2.jsonl
python3 -c "import json;R=[json.loads(l) for l in open('xv2.jsonl')];S={'multi.three_apps','multi.app_switch','multi.return_after_detour','system.spotlight.find_app','safari.read.heading','settings.about.serial','notes.search','notes.create_titled','settings.search.wifi','settings.search.bluetooth','settings.search.storage'};x=[r for r in R if r['task_id'] in S];print(sum(r['passed'] for r in x),len(x))"   # 66 66

The six pairs we marked compatible, and the AgentHorizon type in the table, are our reading of the instructions. Everything else follows from the files.

What would prove this wrong

Our claim is that 40 swapped finishes pass phoneshell's checks. It is wrong if any of them fails on the phone. The cheapest test is multi.app_switch graded on the end state of a calculator.arithmetic run, and notes.search graded after a fresh notes.create_titled run. By 31 December 2026 we expect phoneshell to grade the path for the 8 path tasks from its action log, and to add answer checks to safari.read.heading and settings.about.serial. Rerun after those changes, the implication test should report 6 pairs, all compatible. For AgentHorizon we expect, by 30 June 2027, no judge on the AH split above 90% balanced accuracy with both class accuracies above 85%.

Sources

• Xing Han Lù, Dheeraj Vattikonda, Sina Hajimiri, Fatemeh Pesaran Zadeh, Parishad BehnamGhader, Ghazwa Darwiche, Amirhossein Kazemnejad, Christopher Pal, Alexandre Drouin, Siva Reddy, "AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks", arXiv:2610.11050, 8 October 2026; code and data at github.com/ServiceNow/agenthorizon.
• Blolabel, phoneshell: environments/*.yaml, phoneshell/bench/verify.py, phoneshell/perception/tree.py, read 9 October 2026.
• Blolabel, phoneshell-bench, results/xv2.jsonl, run of 9 September 2026 on an iPhone 14 Pro, iOS 26.3.1.
• Blolabel, "Agents report success, the device disagrees", 15 September 2026, and "12 checks pass before the agent picks", 7 October 2026.