DailyDroid's screenshot adds about 5 points of average success on two models; UI Retrieval, the single largest failure cause, stays the same count with or without it

DailyDroid runs GPT-4o and o4-mini on 75 everyday Android tasks, once with text-only screen state and once with the same text plus a screenshot. Average success moves from 26.7% to 32.0% for GPT-4o and from 29.3% to 33.3% for o4-mini, a gain of about five points either way. The System-Level failure category that causes the most losses, UI Retrieval, has the exact same count in both conditions for both models. The screenshot's entire measured effect lands somewhere else.

Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots (arXiv:2604.17817), submitted 20 April 2026 by Shiquan Zhang, Tianyi Zhang, Le Fang, Simon D'Alfonso, Hong Jia, and Vassilis Kostakos, and published in the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, builds DailyDroid: 75 tasks spanning five everyday scenarios across 25 Android apps, at three difficulty levels. Each task runs twice per model, once given only the Android accessibility tree as text ("screentext") and once given that text plus a screenshot ("multimodal"), on GPT-4o and o4-mini, for 300 runs total. The paper's own framing is blunt about what it expected to find: that screenshot access, which carries a privacy cost and a token cost, "may be unjustified when agents could achieve comparable utility with less invasive representations."

The average gain is five points, not the hundred phoneshell measures on its pixel-only tasks

Across all 75 tasks, GPT-4o succeeds on 20 of them text-only (26.7%) and 24 multimodal (32.0%), a gain of 5.3 points. o4-mini succeeds on 22 text-only (29.3%) and 25 multimodal (33.3%), a gain of 4.0 points. The paper's own summary calls this "comparable performance with multimodal inputs yielding marginally higher success rates," and the numbers support that description exactly: the screenshot helps, consistently, in both models, by a small amount. Neither model's overall rate breaks 34% in either condition, and DailyDroid does not report a breakdown by its three stated difficulty levels, so there is no way from this paper alone to see whether the five-point average gain is concentrated in easy tasks, hard ones, or spread evenly.

The gain lands on Agent-Level prediction errors, not on finding the element

DailyDroid's failure handbook splits every failed run into two tiers: System-Level failures, which the paper attributes to the phone's own UI (retrieval, parsing, logic, execution), and Agent-Level failures, which it attributes to the model's choices (prediction, reflection, reaching the step limit, misjudging an impossible task). The System-Level tier, and specifically its largest subcategory, UI Retrieval, is where the screenshot does nothing measurable: GPT-4o has 25 UI Retrieval failures (33.3%) in both the text-only and multimodal conditions, and o4-mini has 23 (30.7%) in both. Summed across all four System-Level subcategories, GPT-4o's System-Level failure count is 32 (42.7%) in both conditions, and o4-mini's is 30 (40.0%) in both. The entire five-point average gain is produced on the Agent-Level side: GPT-4o's LLM Prediction failures drop from 10 to 7, and o4-mini's drop from 7 to 3, when the screenshot is added.

What it means for reading any "vision helps" or "vision doesn't help" number

• An average gain over a task mix is a statement about the mix, not about the screenshot. DailyDroid's 75 tasks are everyday, mostly tree-answerable Android actions; phoneshell's houseFacts figure isolates tasks where the answer exists only as pixels. The same screenshot that is worth 100 points on one suite is worth five on the other, because the suites measure different things.
• A failure category that does not move with the input is evidence the input was never the bottleneck for that category. UI Retrieval failing at the same rate with or without a screenshot means the model is not failing to see the element; whatever it is failing to do, pixels do not fix it.
• o4-mini's Reaching Max Step failures rise from 3 to 7 when the screenshot is added, the one number in the table that moves against the trend. A benefit on one metric can carry a small, specific cost on another, and an average masks that too.

Check it yourself

# DailyDroid's paper and task/failure tables
curl -s https://arxiv.org/abs/2604.17817
# full tables (success rates, failure taxonomy): https://arxiv.org/html/2604.17817

# phoneshell's pixel-only task definition and its binary result
grep -rl "pixel" phoneshell/bench/*.py environments/*.yaml 2>/dev/null
cat phoneshell/bench/verify.py | sed -n '1,20p'
# the screenshot cost figure (about 35% per step) is in the published run stats:
python3 -m phoneshell.bench.stats --help

What would prove this wrong

We report DailyDroid's System-Level UI Retrieval failure counts as identical between the text-only and multimodal conditions, 25/75 and 25/75 for GPT-4o, 23/75 and 23/75 for o4-mini, read directly from the paper's failure taxonomy table. That claim is wrong if a future revision of arXiv:2604.17817 restates that table with different per-condition counts, or if a follow-up study on a different task mix finds UI Retrieval failures that do move with screenshot access. We also claim phoneshell's own pixel-only result and DailyDroid's average result are not in tension, because they measure different task populations; that reconciliation is wrong if someone runs DailyDroid's exact 75 tasks through phoneshell's grading and finds the pixel-only subset within it behaves like the aggregate rather than like phoneshell's isolated cases. We will check both by 31 March 2027.

Sources

• Zhang, Zhang, Fang, D'Alfonso, Jia, Kostakos, Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots, arXiv:2604.17817, submitted 20 April 2026, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 37, no. 4, read 18 October 2026.
• Full tables: arxiv.org/html/2604.17817 (success-rate table and failure taxonomy table).
• Blolabel, phoneshell: houseFacts and phoneshell/bench/verify.py, read 18 October 2026.