OSWorld 2.0 scores its best agent at 20.6% inside a 500-step ceiling, averaging 318 tool calls a task; phoneshell's hard cap stops at 60, a budget never tested against a task this long

OSWorld 2.0 built its benchmark because the first OSWorld had saturated: agents were clearing it in roughly 30 tool calls. Its answer was to make tasks that take a human a median of 1.6 hours and a frontier model an average of 318 tool calls, and even then the best agent completes one in five. phoneshell's own hard cap on a task is 60 steps, a number no task of its 80 has ever needed to approach.

OSWorld 2.0 ("Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks," arXiv:2606.29537), posted by the XLANG Lab with Snorkel AI on 28 June 2026 and revised 13 July, replaces OSWorld's saturated leaderboard with 108 long-horizon workflows built from everyday and professional computer use. Its own abstract reports that human testers take a median of about 1.6 hours per task, and that Claude Opus 4.7, run with maximum thinking, averages 318 tool calls to attempt one, against roughly 30 tool calls per task on the original OSWorld. Scored under the benchmark's primary metric, a run capped at 500 steps, the best-performing configuration, Claude Opus 4.8 with maximum thinking and batched tool calls, completes 20.6% of tasks outright and earns a 54.8% partial-credit score. GPT-5.5 is reported as more token-efficient per step but plateaus near 13% in the arXiv abstract; the benchmark's own project page gives a close but not identical figure, near 14%, for the same model, and we could not reconcile the two from what either page states.

What changed, and when

The original OSWorld had become easy enough that it stopped discriminating between frontier agents, the same saturation problem that pushed Android benchmarking from AndroidWorld to MobileWorld. OSWorld 2.0's fix was to lengthen the task, not just add more of them: 108 workflows chosen for multi-hour, multi-application real work, each one scored to a 500-step budget instead of whatever a model happened to need. That design choice is itself the finding worth reporting, independent of which model currently tops the board: a benchmark's step ceiling is now explicitly part of what is being measured, set ten times higher than the model that needs it most actually uses on average.

phoneshell's step accounting is set once, in two files. phoneshell/bench/schema.py gives every Task a default of max_steps: int = 25 unless its YAML file overrides it, and a grep across all 80 files in environments/ finds no override. phoneshell/config.py separately sets a global ceiling, commented in the code itself as "Hard cap on agent steps per task, and on money": max_steps: int = 60. Neither number has ever been raised to see what happens, because nothing in phoneshell's task suite has asked for more than the schema default of 25 to begin with.

Two benchmarks' idea of a long task

What it means for someone building or reading a phone-agent benchmark

A step budget is not a neutral measurement instrument; it is a design decision about what kind of task the benchmark is willing to credit as completable at all. OSWorld 1.0 saturated at roughly 30 tool calls per task, so OSWorld 2.0 raised the ceiling to 500 and built tasks long enough to need an average of 318, more than ten times the old average, and the result is a leaderboard where the best model still finishes one task in five. That is the opposite failure mode from saturation: a benchmark hard enough that almost nobody clears it, which is its own kind of uninformative if it persists across several model generations without the top score moving.

phoneshell sits at the other extreme by design, not by accident. Its tasks are built to be scored from device state in a single short session, which is why its schema defaults to 25 steps and its hard cap stops at 60: a task that needed hundreds of steps would not fit the harness's grading model, which checks a specific, narrow claim about the phone's state rather than a multi-hour deliverable. That design choice has a cost phoneshell has not yet measured: nothing in its 80 tasks or its results.json says what would happen to pass rates, cost, or grading reliability if a task were allowed to run for 300 or 400 steps instead of 60. The step ceiling and the task design are co-selected, and changing one without the other is an untested move for either benchmark.

• Comparing success rates across benchmarks with different step ceilings compares the ceiling as much as the model. A 20.6% score at 500 steps and a 91.4% score at phoneshell's own 60-step cap are not evidence that one model is worse; they are scores on tasks built for different time horizons.
• OSWorld 2.0's own jump, 30 to 318 average tool calls between versions of the same benchmark family, shows that a ten-times increase in task length is achievable without the benchmark becoming unusable. phoneshell has made no equivalent jump and has not tested whether its grading approach, built around a short device-state check, would survive one.
• A credible claim that phoneshell's 60-step hard cap is "enough" for its own 80 tasks is still an assumption carried from the schema default, not a measured property, exactly as the earlier MobileWorld step-budget comparison on this site found for a 25-step default against a 50-and-100-step benchmark. This is the same open question at a different scale.

Check it yourself

# OSWorld 2.0's own abstract and task count
curl -s https://arxiv.org/abs/2606.29537

git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "max_steps" phoneshell/bench/schema.py
grep -n "Hard cap on agent steps" -A1 phoneshell/config.py
grep -c "max_steps" environments/*.yaml | grep -v ":0$" || echo "no task overrides max_steps"

# mean turns and seconds across the six models in phoneshell's own board
python3 -c "
import json
d = json.load(open('site/results.json'))
turns = [v['avg_turns'] for v in d['board'].values()]
secs = [v['avg_seconds'] for v in d['board'].values()]
print('mean avg_turns', sum(turns)/len(turns))
print('mean avg_seconds', sum(secs)/len(secs))
"

What would prove this wrong

• We claim no phoneshell task has ever been run with max_steps raised past the schema default of 25, let alone past the hard cap of 60. That is wrong the moment a future run sets a higher value in any environment YAML and results.json records a score for it. We predict that if such a run happened today, most of phoneshell's 80 tasks would still finish inside 25 to 30 steps regardless of the higher ceiling, because they are built to be short, not because the ceiling was binding. We will treat a future run that raises max_steps for any task, and finds that the extra budget changes the outcome, as grounds to revise this prediction, checked by 31 December 2027.
• We note OSWorld 2.0's GPT-5.5 figure differs between its arXiv abstract (near 13%) and its project page (near 14%), and we treat that one-point gap as unresolved rather than picking a side. If a future revision of either page reconciles the number, that resolves this specific uncertainty without affecting the rest of the comparison.