ScaleWoB's five mobile agents average 27.92% against a human 92.08%, and the gap nearly doubles on long-horizon tasks to a 74-point spread; phoneshell has never put a human number next to its own 80 tasks

ScaleWoB's five tested mobile agents average a 27.92% success rate against a measured human success rate of 92.08% on the same 120 tasks, a 64-point gap that widens past 70 points once the task requires more than a few steps. phoneshell has run 120 paired comparisons between frontier models on its own 80 tasks, but it has never run a human through those same tasks to put a number on the other end of the scale.

ScaleWoB, posted to arXiv as 2605.25160 and revised on 18 June 2026, is a framework for synthesizing interactive GUI environments as backend-free webpages with verifiable rewards, built to scale past what hand-built simulators and real third-party apps can offer across mobile, desktop and in-vehicle interfaces. Its released mobile component is a 120-task benchmark spanning 63 simulated mobile applications. Against it, the paper evaluated the five highest-scoring published mobile GUI agents and measured a human success rate on the same tasks for comparison: 92.08% for humans, 27.92% averaged across the five agents, and 17.82% for the agents on the benchmark's long-horizon subset specifically.

What changed, and when

The 92.08% human figure is not a ceiling pulled from a different benchmark; it was measured on the identical 120 tasks the five agents were run against, by the same paper, which is what makes the comparison meaningful rather than a borrowed number. The agent average of 27.92% sits 64.16 points below that, and the drop to 17.82% on the long-horizon subset means the gap on the hardest tasks in the set is closer to 74 points than 64. phoneshell's own published results compare models against each other (120 paired runs across 3 frontier models, pre-registered, with the screenshot-ablation result as its headline) and report pass rates per model on its 80 tasks. Nowhere in the public results.json or the site's houseFacts is there a human operator's own completion number on those same 80 tasks to anchor what a given pass rate is actually close to or far from.

A baseline row phoneshell's own tables leave blank

A human baseline measured on the exact tasks a model is scored against turns a pass rate from a number that only means something relative to other models into one that also means something relative to how hard the task actually is for a person. ScaleWoB's 27.92% reads very differently next to a 92.08% human ceiling than it would read alone. phoneshell's site structure (the leaderboard and results.json referenced throughout its published pages) is built entirely around per-model comparisons: pass_rate, avg_turns, avg_seconds and total_cost_usd, one row per model. There is no row, and no field in results.json's schema, reserved for a human operator running the same 80 tasks once each and reporting what fraction they complete, how many turns they take, or how long they take, the three numbers that would let a reader place phoneshell's own model results on the same kind of scale ScaleWoB's chart makes possible.

What it means for anyone publishing an agent leaderboard

• A pass rate without a human anchor answers "which model is better" but not "how far is any of them from done." ScaleWoB's chart makes both questions answerable from the same table; a leaderboard with only model rows, phoneshell's own included, can only answer the first.
• Measuring a human baseline costs something real: a person's time on every task in the set, at least once. That is a judgment call about where to spend effort, not an oversight by default, and ScaleWoB's own baseline is a single human run per task, not a sample large enough to report a human variance the way the agent average is reported across five models.
• phoneshell's difficulty field (easy, medium, hard) is a guess at how hard a task is for a model, made by whoever wrote the task, not a measurement. ScaleWoB's long-horizon-versus-full-set split is closer to a measured property: it is defined by how the agents' own scores behave, not asserted up front.
• Adding a human-baseline row to phoneshell's results.json would not require a new task schema field, since difficulty, max_steps and checks all apply unchanged to a human run; it needs a person to actually sit down and run the 80 tasks once, which has not happened yet.

Check it yourself

# ScaleWoB's human-vs-agent numbers
curl -s https://arxiv.org/abs/2605.25160 | grep -i "92.08\|27.92\|17.82\|human"
# project site: https://scalewob.github.io

curl -s https://blolabel.ai/results.json | python3 -c "import json,sys; d=json.load(sys.stdin); print(list(d['board'].keys())); print('human' in str(d).lower())"

What would prove this wrong

We claim phoneshell's results.json and public houseFacts contain no human-baseline row for its 80 tasks. That is wrong the moment a human-run row appears in results.json's board or runs sections; we found none as of 24 October 2026. We also report ScaleWoB's human baseline as 92.08% on its own 120-task set, measured once per task rather than across a sample; that figure is wrong if a later revision of arXiv:2605.25160 reports a different human success rate or a variance range, which we will check by 30 April 2027.