How accurate are LLM judges on OSWorld-Pro?

Against 67,264 human step labels, the best judge, GPT-5.6-Sol at max reasoning, reaches 97.0 macro-F1 on which subgoal a step targets and 94.3 on completion, but 74.7 on progress and 61.9 on feasibility. Human-to-human agreement on feasibility is 91.6, Cohen's kappa 0.847.

Does more reasoning help an LLM judge?

On most fields, yes. On feasibility, no: Sol falls from 67.5 with reasoning off to 61.9 at max, and Terra from a peak of 67.4 at medium to 62.0 at max. Sol labels 9.7% of steps infeasible with reasoning off and 5.1% at max, against a human rate of 15.3%.

Why does feasibility matter for grading agents?

A step toward something that does not exist is an environment failure. A judge that calls it feasible charges the failure to the agent. At a No rate one third of the human rate, the judge moves failures from the environment's account to the agent's.

How does phoneshell decide a task was never attempted?

In code, before grading. A locked or unresponsive phone raises RigUnavailable and a rate-limited account raises UsageLimitReached; either way the task is reported as not scored. An early run without this rule recorded 12.5% when 27 of 32 tasks were never attempted.

← Notes

analysis 23 September 2026 5 min read

More reasoning made OSWorld-Pro's judge worse at spotting steps that could not succeed

GPT-5.6-Sol at maximum effort agrees with humans on feasibility at 61.9 macro-F1, against 91.6 between humans, and calls a step infeasible a third as often as they do.

The judge that scores OSWorld-Pro agrees with human annotators on whether a step was even possible at a macro-F1 of 61.9, against 91.6 between humans, and it gets worse the harder it thinks. GPT-5.6-Sol labelled 9.7% of steps infeasible with reasoning off and 5.1% at maximum effort; the humans labelled 15.3%. OSWorld-Pro, arXiv:2609.24890, was posted on 21 September 2026 and uses the maximum-effort setting as its grader.

Abstract technical illustration of a row of doorways, a few of them sealed, with an inspecting beam that brightens from left to right while the sealed doorways it marks become fewer

What OSWorld-Pro added

Zhilin Wang and 16 co-authors, Jan Kautz among them, annotated 305 desktop tasks broken into 2,814 ordered subgoals, with 67,264 step-level human labels. For every step of an agent's trajectory the annotators record four things: which subgoal the step targets, whether that subgoal is feasible in the environment as it stands, whether the step makes progress, and whether it completes the subgoal. The guideline is strict about the order: if feasibility is No, progress and completion must be No as well. An independent reviewer relabelled the same steps to measure agreement: Cohen's kappa was 0.987 for target, 0.847 for feasibility, 0.869 for progression and 0.940 for completion.

Hand-labelling 67,000 steps does not scale to new models, so the paper tests three GPT-5.6 judges at six reasoning efforts each, then scores the leaderboard with the best one. The headline results are Claude Opus 4.8 Max at 77.7% of tasks with every subgoal complete and Claude Opus 5 Max at 75.7%, against 83.4% for Opus 5 on the original OSWorld. The judge table underneath it matters more to anyone who grades agents with a model.

Labeller	Target	Feasible	Progress	Complete	Step mean	Est. cost, full run	Source
Human annotators	98.4	91.6	94.1	97.6	95.4	not reported	Table 1
GPT-5.6-Sol, max	97.0	61.9	74.7	94.3	82.0	$249.60	Table 1
GPT-5.6-Sol, none	94.5	67.5	69.5	67.8	74.8	$188.99	Table 1
GPT-5.6-Terra, max	97.6	62.0	69.2	94.8	80.9	$155.04	Table 1
GPT-5.6-Terra, medium	94.9	67.4	59.6	82.9	76.2	$98.89	Table 1
GPT-5.6-Luna, max	97.7	58.9	52.8	93.3	75.7	$15.45	Table 1
GPT-5.6-Luna, low	94.9	54.0	46.4	87.0	70.6	$9.66	Table 1

Macro-F1 against the human labels, from Table 1 of arXiv:2609.24890. Cost is the paper's estimate for judging the full benchmark once at list API prices without caching. Seven of the eighteen judge rows are shown; the chart below carries every feasibility value.

Feasibility is the one column that reasoning does not fix

Going from no reasoning to maximum effort, Sol's completion F1 rises from 67.8 to 94.3. Its feasibility F1 falls from 67.5 to 61.9. Terra peaks on feasibility at medium effort, 67.4, and ends at 62.0. The paper's diagnosis is in its own numbers: with more reasoning the judge says No less often, 9.7% of steps at Sol none and 5.1% at Sol max, while the human ground truth is 15.3%. The judge talks itself into believing a step could have worked. Luna does not follow the pattern, ranging from 54.0 at low effort to 60.4 at xhigh, and the paper states the trend only for Terra and Sol.

A judge that over-believes feasibility charges the environment's failures to the agent

The feasibility label decides which column a failure lands in. A step toward a menu option that does not exist in that distribution is, correctly labelled, infeasible, and the resulting miss belongs to the environment. Mislabelled as feasible, the same step becomes a subgoal the agent failed to progress. A judge that says No a third as often as humans (5.1% against 15.3%) moves failures from the environment's account to the agent's. The paper gives the reason it is hard: a menu option can look absent until a sub-menu is expanded, so feasibility is partly a claim about what the agent did not try.

We have our own number for how large this effect can be. The docstring on UsageLimitReached in phoneshell/bench/runner.py records that an early run "recorded 12.5% when in truth 27 of 32 tasks were never attempted". Nothing about the model's capability produced that 12.5%. A judge asked "was this possible" would have seen 27 empty or truncated trajectories and had to guess; the runner instead sees the rate-limit message, raises, and the task is reported as not scored. Same for RigUnavailable on a locked or unresponsive phone.

What to do with this if you grade with a model

Take feasibility away from the judge wherever the environment can answer it. On a phone that covers more than it sounds like: the app is installed or not, the account is signed in or not, the network answered or not, a permission sheet blocked the screen or not. Each is a fact the runner can check before or during the run. What is left for the judge is the ambiguous remainder, where the paper's menu example lives.

Do not buy judge accuracy with reasoning effort on this column. Sol max costs $249.60 per full evaluation against $188.99 with reasoning off, and on feasibility it is 5.6 points worse. If you need a model to call feasibility, measure its No rate against a human sample first; a judge whose No rate is a third of the human rate will inflate agent failure in exactly the tasks where the environment is at fault.

And report the No rate. OSWorld-Pro's paper is unusually good here because it published per-field agreement and the No-rate drift. Most agent papers publish one judge-agreement figure, which in this table would be the step mean of 82.0 and would hide the 61.9.

Check it yourself

curl -sL https://arxiv.org/html/2609.24890 -o oswpro.html
python3 - <<'PY'
import re, html
t = html.unescape(re.sub(r'<[^>]+>', ' ', open('oswpro.html').read()))
t = re.sub(r'\s+', ' ', t)
for key in ('feasibility=No decreases', "Cohen", '61.9 vs. 91.6'):
    i = t.find(key); print(t[max(0, i-200):i+400], '\n---')
PY
# In this repository:
grep -n "27 of 32" phoneshell/bench/runner.py
grep -n "class RigUnavailable\|class UsageLimitReached" phoneshell/bench/runner.py

Table 1 in the HTML edition comes out as one value per cell. Each judge row reads, in order: Target, Feasible, Progress, Complete, Step mean, Subgoal, Task (1-MAE), input tokens per task, output tokens per task, Total $.

What would prove this wrong

The claim is that reasoning effort pushes LLM judges toward calling impossible steps possible, and that this charges environment failures to agents. It is wrong if the OSWorld-Pro judge prompt, rerun by anyone at Sol max with an explicit instruction to prefer No when an option cannot be found after exploring the relevant menus, reaches a feasibility F1 above 80 and a No rate within 3 points of the human 15.3%. It is also wrong for phones if we run a model judge over our own trajectories and it recovers every one of our code-decided not-scored tasks from the trajectory alone. We will run that second test on the physical-iPhone suite and publish the confusion matrix by 31 December 2026.