Five computer-use agents on blind users' real commands: success spreads 14.6 points, partial completion stays at a third

GPT-5's lead over Claude Sonnet was not significant (p = .071). The more useful number is the one that did not move: a third of commands stalled part way for every model, most often at the end, where the agent decides it is done.

In a three-week field study of computer-use agents with blind users (arXiv:2609.00524, 1 September 2026), five models ran the same 1,258 real commands and their success rates spread from 37.9% to 52.5%, while their partial-completion rates stayed between 33.3% and 34.9%. A better model moved commands from outright failure to success. It did not shrink the third of commands that stalled part way. Our reading of the paper's own table: the model is not what decides that third.

The study, and what it measured

Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou and I. V. Ramakrishnan (Stony Brook University) with Vikas Ashok (Old Dominion University) posted the paper on 1 September 2026. Eight blind screen-reader users installed a screen-reader-accessible agent called OLLA on their own Windows machines for three weeks and issued 1,258 commands, normalised to 304 distinct task intents, across 12 applications: Word, Excel, PowerPoint, OneNote, Outlook, File Explorer, VLC, Windows Media Player, Zoom, Calculator, Notepad and Spotify.

GPT-5 ran live during the deployment. The authors then replayed every command against four more models (Claude Sonnet, Gemini 2.5 Computer Use, UI-TARS and Qwen3-VL) from a fresh application state, with the prompt, action schema and executor held fixed. At every step the agent received both a screenshot and the Microsoft UI Automation tree. Four annotators labelled each trace success, partial completion or failure against reference steps they had performed themselves, with Krippendorff's alpha of 0.84.

The paper reports that GPT-5's 4.0 point lead over Claude Sonnet is not significant in a paired McNemar test (chi-squared 3.25, p = .071), and says so plainly. It also notes that partial completion "ranged from 33.3% to 34.9%". The last row is ours: the spread in success (14.6 points) is almost exactly the spread in failure (15.6 points), and the partial band barely moves.

Where the stalled third breaks

The partial traces were not near-misses at the first step. GPT-5's partial attempts completed 68.3% of the reference steps on average before breaking down, and even Qwen3-VL's completed 53.0%. The paper's failure analysis locates the late breaks. Termination recognition, where the agent stopped too early, repeated an action without noticing nothing was changing, or never emitted done, accounted for 17.6% of GPT-5's partial attempts (76 of 431) and 19.1% across all models (410 of 2,148). One Excel trace created a new sheet and then kept pressing the new-sheet control. A Word trace chose PDF in the export path and never pressed Save.

What it means for anyone building or ranking a phone agent

Do not rank on the headline. At 1,258 paired commands, the top two models are not separable (p = .071), which the authors report. We found the same on a phone: six models over 58 tasks on one iPhone and not one pair separated. A paired test belongs next to every leaderboard number, and the per-outcome split belongs next to the pass rate.

Budget loop engineering for the stalled third. Our judgement from the flat partial band: the failures that swapping models fixes are the ones at the front of a task, and the ones it leaves are the late ones the paper names, termination, commit steps, and constraints carried across steps. Those are loop design problems: a check that the screen changed after each action, a stop rule for repeated actions, and a final-state check before done is believed. That is a hypothesis from one table; the paper does not test it.

Treat done as a claim. In OLLA, completion is the model emitting done, and a blind user hears that, often without a cheap way to verify it; the paper notes that participants wanted agents to confirm before hard-to-undo actions. On a phone we measured how often that signal is wrong: 16 of 27 failures were reported as successes by the model that failed them. For an assistive agent the self-report is the user's only instrument, so grading from device or application state is part of what the user is buying.

The paper is a Windows desktop study with UI Automation and screenshots together, and it does not separate the two inputs. It says nothing direct about iOS. What transfers is the shape of the loop and the failure categories; the percentages stay with Windows.

Check it yourself

Everything above comes from the paper's HTML version and its public artifacts. The spreads are one line of arithmetic.

# the outcome table (Appendix C, Table 5) and the step-progress table (Table 1)
curl -sL https://arxiv.org/html/2609.00524 -o olla.html
python3 - <<'EOF'
import re, html
t = html.unescape(re.sub(r'<[^>]+>', ' ', open('olla.html', encoding='utf-8').read()))
t = re.sub(r'\s+', ' ', t)
i = t.find('Appendix C Task Outcomes'); print(t[i:i+520])
EOF

# the spreads
python3 -c "s=[52.5,48.5,43.9,39.8,37.9]; p=[34.3,34.1,34.9,34.2,33.3]; f=[13.2,17.4,21.2,26.0,28.8]
print(round(max(s)-min(s),1), round(max(p)-min(p),1), round(max(f)-min(f),1))"
# 14.6 1.6 15.6

# the agent itself, and the traces
open https://github.com/Satwikram/OLLA

The anonymised dataset of commands, model outputs and annotated traces is linked from the paper's section B.4. Per-application outcome counts are in its Table 6 if you want to test whether the flat partial band holds inside each of the 12 applications or only in aggregate.

What would prove this wrong

• Re-run the released commands with a newer model and see partial completion fall well below 33%, for example under 25%, while success rises. That would mean the flat band was a property of these five models rather than of the loop. We predict that any model evaluated in this setup before 30 June 2027 lands between 28% and 40% partial.
• Break the band down by application from Table 6 and find it is flat only in aggregate, with the per-application partial rates moving as much as success does. Then our reading is an averaging artifact.
• Add a final-state check and a repeated-action stop rule to OLLA's loop and see no change in partial completion. That would falsify the claim that the stalled third is a loop-design problem.