Agents' 'done' signals miss in both directions: on WebPageBench one framework over-claims by 41 points and another hides a fifth of its real successes

WebPageBench grades 24 model and framework pairs from the site's own event log and records the agent's completion claim separately. The claim over-counts by up to 62 of 152 tasks on some frameworks and under-counts by up to 21 on others, so no single correction fixes self-reported success.

On WebPageBench's public leaderboard, gpt-5.6-luna running in the OpenManus framework declared all 152 tasks finished, and the site's own event log confirmed 90 of them: a 41-point over-claim. The OpenHands framework errs the other way on every model it runs, leaving 74 of its 383 log-verified successes (19%) without a completion signal. Self-reported success is not one bias you can subtract. Its sign depends on the framework around the model.

What WebPageBench released on 28 September 2026

WebPageBench (arXiv:2609.35026, v1 dated 28 September 2026) is from Anton Emelyanov, Maria Tikhonova, Zaven Martirosian, Sergei Averkiev and Alena Fenogenova (DAIMLD, HSE University, NUST MISIS). It is six mock sites with brand cues removed: a marketplace, a bookstore, grocery delivery, rail tickets, hotel search and a document cabinet. The pages themselves write typed events (19 event types, such as basket_add with item id and quantity) to a per-run log. A task lists the events it needs; the Event-Match Score (EMS) is 1 only if every condition matches. No judge model is involved and the rendered page is never scraped.

The suite has 152 tasks: 65 hand-written scenarios plus 87 variants that re-render one control (a popup calendar becomes a native date field, say) with the prompt and conditions unchanged. The agent's own completion claim is stored separately from EMS, and a third number, Done&pass, counts tasks where the agent said done and the log agreed. The leaderboard covers 24 model and framework pairs, one run each at default decoding. The code is on GitHub at ai-forever/WebPageBench (repository created 23 September 2026, Apache-2.0) and the leaderboard is a Hugging Face Space.

Over-claims are (said done minus Done&pass) x 152 and under-claims are (EMS minus Done&pass) x 152, rounded; the paper publishes the three rates, not these counts. Observation types are from the paper's Table 1; the DeepSeek pairs receive DOM text only.

The framework decides which way the self-report is wrong

In 8 of the 24 pairs the agent said done on 151 or 152 of 152 tasks. Those eight are Gemini and GPT-5.6-luna on openmanus and browser-use, DeepSeek on browser-use, and Qwen3.8-27B on openmanus, ouroboros-cut and browser-use, and their verified scores run from 0.592 to 0.822. A completion rate of 1.000 carries no information about which tasks worked. The same GPT-5.6-luna under OpenHands is nearly calibrated, 0.638 said done against 0.625 verified.

OpenHands flips the sign. It under-claims on all five models it runs: Gemini 18 of 87 verified successes, DeepSeek 18 of 69, GLM-5.2 14 of 76, MiniMax 21 of 56 and GPT-5.6-luna 3 of 95, which is 74 of 383 pooled. The other DOM and text frameworks together leave 5 of 1,227 verified successes unreported. The screenshot-only agents do both at once: 158 over-claims and 59 unreported successes out of 457 verified, 12.9%. The paper's own summary for that group is that between one in nine and one in four verified successes end without a completion signal.

The completion-minus-EMS gap across the 24 pairs runs from minus 11.2 points (DeepSeek on OpenHands) to plus 40.8 (GPT-5.6-luna on OpenManus). A grader who calibrates self-reports on one framework and applies the correction to another would be wrong by up to 52 points. These are single runs per pair, so a two- or three-task difference in one row is noise. A consistent sign across five models on one framework is not.

Same model, same prompts: DOM 0.763, screenshots 0.480

Qwen3.8-27B scores 0.737 to 0.763 through three DOM frameworks and 0.480 through the screenshot-only adapter, 116 against 73 of the same 152 tasks. The screenshot adapter returns pointer coordinates through the authors' Playwright driver; the DOM frameworks act on element handles. This is the web version of the tree-versus-pixels question our own benchmark asks on iOS, and on this suite the element tree wins by 43 tasks for one model. One run, one model, so treat the size as an anecdote; the direction matches every other screenshot agent on the board, all of which sit at or below 0.638.

What it means for grading an agent on a phone

Our own phone data sits in the over-claim regime. In the 9 September 2026 sweep behind our 15 September note, 16 of 27 device-graded failures ended with the model saying the job was done, and none of the 441 device passes across two sweeps had a summary admitting failure. That matches the DOM frameworks here, and the reason is structural in both. phoneshell's OpenRouter loop ends a run when the model calls phone_done, and a reply with no tool call is taken as the answer (phoneshell/bench/openrouter.py), so a completion signal fires on almost every run by construction, the way browser-use's done action does.

The two benchmarks disagree about the under-claim cell, and it is a decision a grader has to make on purpose. WebPageBench scores a task from the log whether or not the agent said done, unless the shared time limit expires. phoneshell refuses to score a pass for a run that hit its 240-second budget or its 25-turn cap, whatever the phone shows afterwards (result.passed = passed and not timed_out in phoneshell/bench/runner.py). That rule came from a measured case: clock.alarm.set_time once timed out with zero completed turns and still satisfied its checks from what was left on screen. Our judgement is that on a real phone the stricter rule is right, because an agent that keeps acting after the goal is met can undo it. The paper does not break its under-claims down by cause (step cap, framework error, or a clean stop without the done action), so we cannot say how many of OpenHands' 74 would fail under our rule.

We also cannot measure our own under-claim rate the way WebPageBench does. phoneshell stores the model's final text and any timeout error, but not whether the text came from phone_done or from a plain reply. Adding that one field would make both cells of the bottom row countable.

Check it yourself

• Paper: https://arxiv.org/abs/2609.35026; Table 3 (24 pairs: EMS, Compl., Done&pass), Table 1 (observation types), Section 5.2.
• Counts: for each row, over = round(152 x (Compl. minus Done&pass)) and under = round(152 x (EMS minus Done&pass)). Gemini on OpenHands: 152 x (0.572 minus 0.454) = 17.9, so 18.
• Code and leaderboard: https://github.com/ai-forever/WebPageBench, https://huggingface.co/spaces/ai-forever/WebPageBench.
• phoneshell: grep -n "No tool call and no phone_done" -A 3 phoneshell/bench/openrouter.py; grep -n "passed and not timed_out" phoneshell/bench/runner.py; grep -n "max_steps\|timeout_seconds" phoneshell/bench/schema.py.

What would prove this wrong

Dated 31 March 2027. If WebPageBench or anyone else re-runs the OpenHands framework on the 152 tasks with at least three of its five models, said-done will again be below EMS on at least two of them. If said-done comes out at or above EMS on all three, the OpenHands under-claim is noise from one run. On our side: once phoneshell records where the final summary came from, runs that pass on the device without a phone_done call will be under 2% of device passes over the next 300 scored runs. A higher share would mean our over-claim-only picture from September missed a whole cell.