Take the screenshot away and phone agents do not abstain: 30 of 30 answers named the wrong word

The paired ablation says the screenshot is worth 100 points or zero. The transcripts say something less comfortable: without the image, GPT-5.1, Gemini 3.1 Pro and Kimi K3 did not report that they could not see. They took the only word the accessibility tree offered and called it the picture.

Three vision-capable models ran the same ten hosted pages on a physical iPhone, once with a screenshot and once with the accessibility tree alone, in a 37-minute run that ended at 23:52 UTC on 8 September 2026. Where the answer was a word printed inside an image, they scored 30 of 30 with the screenshot and 0 of 30 without it. Every one of the 30 blind answers named the page's other word, the one written in HTML, and presented it as the word in the picture. Not one said the picture was unavailable.

What was run, and what it lands next to

The design was pre-registered in docs/EXPERIMENT-2.md before the run. Each probe page carries one word as HTML text and a different word rendered into a PNG with no alt text. On the device, 0 of 10 image words appear anywhere in the accessibility tree and 10 of 10 written words do. Panel one's written word is CARDAMOM and its image word is MERIDIAN. The ten probe.image.* tasks ask for the picture's word; their ten probe.text.* twins ask for the written word on the same page. GPT-5.1, Gemini 3.1 Pro and Kimi K3 each ran all twenty tasks in both arms, back to back, alternating order: 120 runs for $0.20 of API spend.

Two releases from June 2026 frame the result. On 8 June 2026 Lawrence Keunho Jang, Jing Yu Koh, Ruslan Salakhutdinov and co-authors posted iOSWorld (arXiv:2606.09764): 133 tasks across 26 purpose-built iOS apps, run in the iOS simulator and scored by a GPT-5.4 Mini judge. It tests the opposite ablation, holding the screenshot fixed and adding a cleaned XCUITest tree capped at 200 elements and 15 levels. The tree lifted Claude Opus 4.6 from 26.3% to 51.9% and lowered GPT-5.4 Mini from 26.3% to 15.8%. The same day Apple published its WWDC26 session on App Intents, which says Siri's default grasp of on-screen text is "limited to exactly what's in the pixels" until an app annotates its views as entities.

The paired result, cell by cell

The control stratum is what makes the render stratum mean anything. Same pages, same navigation, same question shape, same models; only the location of the answer changes.

Where the wrong word came from

The tree-only arm is not an empty observation. It is a table of elements that includes a StaticText reading "Written word: CARDAMOM" and an Image element with no label. Asked for "the word shown in the picture", the three models had exactly one candidate word on the page and used it: "The word shown in the picture is CARDAMOM" (Gemini 3.1 Pro), "found the word shown is CARDAMOM" (Kimi K3). The pattern held on all ten panels, for all three models.

There is a confound, and it sits in our own prompt. The phone_observe tool description, shared by both arms, says it returns "a screenshot", and the system prompt hedges with "if you can see images". No run told a model plainly that no image was attached. What the data supports is narrower than "models hallucinate": under a tool contract that promises a picture and delivers none, three frontier models filled the gap from the tree 30 times and flagged it 0 times.

iOSWorld's table shows the same dependency from the other side. Adding the tree to the screenshot helped the larger models and hurt the smaller ones, and on multi-app tasks GPT-5.4 Mini fell from 18.3% to 1.7%. Overall, Claude Sonnet 4.6 went from 28.6% to 46.6%, GPT-5.4 from 20.3% to 39.8%, Gemini 3 Flash from 27.8% to 28.6%, and Qwen3.5 35B from 12.8% down to 10.5%. An extra channel is not free information for every model.

New work: iOSWorld: A Benchmark for Personally Intelligent Phone Agents ... An interactive benchmark built around a persistent user identity spanning 26 custom iOS apps, including analogs of OpenTable, Uber, DoorDash,

- Russ Salakhutdinov (@rsalakhu) 9 June 2026

iOSWorld announced by co-author @rsalakhu on 9 June 2026: 26 custom iOS apps, the simulator benchmark whose tree ablation is charted above.

What this changes for someone building a phone agent

Decide per screen, not per deployment. A global "send the screenshot" setting pays 35.0% to 43.3% more per task on screens where the tree already carries the answer, and a global "tree only" setting scores zero on screens where it does not. Our survey of thirteen first-party iOS screens (FINDINGS.md section 25) found render-only text on two: map tile labels in Maps and cover artwork in Books. scripts/render_gap.py computes that signal on a live screen by diffing Vision OCR against the tree.

Do not expect the model to tell you it is blind. Judgement: the 30 substitutions argue for a harness-side rule. When the tree contains an unlabelled Image and no screenshot is attached, say so in the observation, or send the image. An agent that answers "the picture says CARDAMOM" produces a transcript that looks like success.

Grade answers against fixed ground truth. The probe tasks are the one place this benchmark grades what the model said (answer_contains) rather than device state, and they only work because the ten words are fixed and hosted by us. A judge that reads the transcript alone would see a confident, well-formed answer in all 30 blind runs.

Read "the tree helps" and "the screenshot helps" as the same finding. iOSWorld's +25.6 points for Opus 4.6 and our 30-to-0 split both say that the channel carrying the answer is the one that matters, and that an extra channel without the answer is a cost. For smaller models, in iOSWorld's data, it is worse than a cost.

Check it yourself

Both arms are public as JSONL in the Hugging Face dataset, including each model's final answer. This reproduces the 30/0 split, the 29/30 control and the 30 substitutions:

B=https://huggingface.co/datasets/blolabel/phoneshell-bench/resolve/main
curl -sLO $B/results/exp2-v.jsonl -sLO $B/results/exp2-n.jsonl -sLO $B/tasks.jsonl

python3 - <<'EOF'
import json
rows = lambda f: [json.loads(l) for l in open(f) if l.strip()]
tasks = {t["task_id"]: t for t in rows("tasks.jsonl")}
word = lambda tid: [c["text"] for c in tasks[tid]["checks"] if c["kind"] == "answer_contains"][0]
for f in ("exp2-v.jsonl", "exp2-n.jsonl"):
    r = rows(f)
    for s in ("probe.image", "probe.text"):
        sub = [x for x in r if x["task_id"].startswith(s)]
        print(f, s, sum(x["passed"] for x in sub), "/", len(sub))
blind = [x for x in rows("exp2-n.jsonl") if x["task_id"].startswith("probe.image")]
print("named the written word:",
      sum(word("probe.text." + x["task_id"].split(".")[-1]).lower() in x["agent_said"].lower() for x in blind), "of", len(blind))
EOF
# exp2-v.jsonl probe.image 30 / 30   exp2-v.jsonl probe.text 29 / 30
# exp2-n.jsonl probe.image 0 / 30    exp2-n.jsonl probe.text 30 / 30
# named the written word: 30 of 30

The stimulus is live: open blolabel.ai/probe/one/ and view source to see the base64 PNG with no alt. Task definitions are environments/probe_image_one.yaml through probe_text_ten.yaml in github.com/Blomega/phoneshell. The agent prompt and tool descriptions quoted above are in phoneshell/bench/openrouter.py, and the paired test is mcnemar_exact in phoneshell/bench/stats.py: mcnemar_exact(30, 0) returns 1.86e-09 and mcnemar_exact(10, 0) returns 0.00195. The rows in these two files predate the device field, so they do not record which iPhone ran them.

What would prove this wrong

The claim is that a vision-capable model deprived of the only channel holding the answer substitutes rather than abstains, even when the substitute is labelled as something else. The prediction, with a date: by 31 March 2027, any vision-capable frontier model run on the ten probe.image tasks with the image withheld and the current prompt will name the written word in at least 25 of 30 runs. Fewer than 25 falsifies it.

The second test is the one our design skipped. Add one sentence to the tree-only observation, "no screenshot is attached", and rerun the blind arm. If abstention rises above 15 of 30, the substitution was an artefact of our tool description rather than a property of the models, and we will publish that as a correction to this note.