WebFovea gained 26 points without changing its model; phoneshell's live agent still gets a 235 x 512 picture of the phone

Jiangang Han's WebFovea report is the cleanest measurement yet of how much of an agent's score lives outside the model. We ran its four-stage checklist over phoneshell's own code. Two of the four stages are fine. The other two carry the same failure shapes the paper measured.

WebFovea, a vision-based web agent, went from 31.0 to 57.0 out of 100 on the WebRetriever Challenge 2026 hidden set in four submissions between 29 August and 2 September 2026, and used claude-opus-4-6 for every one. The largest single fault it fixed was not reasoning: the model API was shrinking screenshots, so every click landed at three quarters of its target. phoneshell cannot make that mistake, because its agent taps element ids, but its live MCP server shrinks the iPhone screen to 235 x 512 pixels on purpose.

What WebFovea changed, and on which date

Jiangang Han, an independent researcher, posted the report as arXiv:2610.03036 on 2 October 2026. The challenge, run by Mininglamp Technology, scores Protocol III of the WebRetriever benchmark (arXiv:2607.06118): start from an entry URL on a live website, operate the site, return one exactly checkable answer, no partial credit, with a manual review of trajectories. Teams got one hidden-set run a day for seven days and an aggregate score only. Models had to come from an approved list, and the newest Anthropic model on it was Claude 4.6. WebFovea placed 2nd.

Each submission shipped several changes, so the paper does not credit the official gains to single components. It does break down the second jump: the 29 August run finished only 79 of 100 tasks before its time limit, at 39.2% accuracy on those. Finishing 97, as the next run did, would have scored about 38, so roughly 7 of the 10 points came from finishing tasks and 3 from answering better. Per-component evidence comes from small paired sets on the 100 public tasks, which the author labels as direction, not significance.

The model was right and the click was wrong

The organisers' template mapped model coordinates with UI-TARS's smart_resize rule, which for a 1920 x 1080 screenshot is nearly the identity. Across 1,312 clicks in the first full run, no click fell right of or below three quarters of the screen. A calibration image with six known markers showed why: the API downscaled the image before the model saw it, and coordinates needed a correction of 1.324 in x and 1.350 in y. Sending 1440 x 810 directly brought the factor to 1.010 and 0.993. On a paired set of 24 tasks the version with this fix went from 8 to 20 passes, 12 fixed and 0 broken, and the share of tasks hitting the step limit fell from 67% to 4%. Three targets on a real page that had been missed by 122 to 189 pixels were hit within 1 to 5.

How the four stages map onto a phone agent

The paper's checklist is model-agnostic: for any failure, ask whether the reply was parsed as meant, whether the action took effect, whether the result was reported truthfully, and whether the model was shown what it needed. phoneshell's design already closes the first two for most actions. Structured tool calls have no free text to contaminate, and the model picks an element by id from the accessibility tree, so the coordinate conversion lives entirely in phoneshell's Geometry class (pixels divided by the device scale, 3 on current iPhones). That is the stage where WebFovea lost the most, and it is a design choice, not luck.

The other two stages are where the paper's findings land on our code. WebFovea's most useful negative result is that a correct "nothing changed" note did not change behaviour: the model acknowledged the note in its own reasoning and repeated the action anyway, across 10 tasks and 400 steps. Only budgets enforced by the executor, which refuse the call with an explanation, stopped the loops. phoneshell's live loop guard is exactly the note: _act_and_report returns "THE SCREEN DID NOT CHANGE ... try something different rather than repeating it". The class built to enforce a limit, StuckDetector in agent/verify.py, has no caller in the package. Our earlier note on the Inertia paper found the same pattern from the other side: a screen-unchanged notice barely stops repeats.

The picture the live agent gets

WebFovea found its observation bug because an outside API was resampling its images. phoneshell resamples its own. Config.brain.screenshot_max_edge defaults to 512, both in config.py and in the runtime config.yaml, and observation.py passes it to downscale() before JPEG encoding at quality 70. An iPhone 17 Pro Max screenshot of 1320 x 2868 becomes 235 x 512, 3.2% of the original pixels. The benchmark runner overrides the value to 1536 (706 x 1535, 28.6% of the pixels), so published bench results do not use the small image, and a person driving the phone through Claude Code or another MCP client does.

We have measured what that size does. phoneshell's own site records that an earlier version of the vision experiment reported vision as worth nothing because "the image had been downscaled to 235×512", and that result was retracted before the pre-registered run that found 100% with the image and 0% without on pixels-only answers. The MCP default is the retracted configuration. Judgement: for a labelled Settings screen it costs nothing, because the tree carries the answer; for a chart, a map tile or text drawn into an image, the live agent is running the experiment we threw out.

What this means if you build or run a phone agent

• Measure the image the model receives, not the one you captured. WebFovea's fix started with a synthetic target of six markers; the phone equivalent is a screenshot with known glyphs at known points, read back through the real path.
• Move loop limits out of the prompt and into the executor. WebFovea's data says a note is read and ignored. A refusal after N identical actions on an unchanged screen is a few lines in _act_and_report.
• Read back after typing even when submitting: read the field before pressing return, the way WebFovea's fill reads back the value and leaves Enter to a separate step.
• Decide whether shortcuts count. The challenge banned goto because answers must come through the site's own interface. phoneshell grades end state, so a deep link that reaches the right screen passes. That is a defensible rule, but a score built on it is not comparable with one that forbids it.
• Budget for live-site drift if you compare against WebFovea: 13 of the 100 public reference answers no longer matched the live sites, and 16 sites were unreachable from the author's network.

Check it yourself

• Scores and dates: Table 4 of https://arxiv.org/abs/2610.03036. Coordinate numbers: Sec. 4.3 and Table 5. Feedback negative result: Sec. 5.4.
• The default: grep -n screenshot_max_edge phoneshell/config.py phoneshell/bench/openrouter.py in the phoneshell repo shows 512 for the live config and 1536 for the bench.
• The arithmetic: python3 -c "r=512/2868;print(int(1320*r),int(2868*r),round(int(1320*r)*int(2868*r)/(1320*2868)*100,1))" prints 235 512 3.2.
• The unused detector: grep -rn StuckDetector phoneshell/ returns only its definition in agent/verify.py.
• The skipped read-back: in phoneshell/actions.py, _confirm_typed starts with if submit: return result.
• The retraction text: grep -n "235×512" phoneshell/bench/site.py.

What would prove this wrong

The claim is that 235 x 512 is too small for pixels-only answers on the live path. Test: run the 10 pre-registered pixels-only pages through the MCP server twice, once at the default 512 and once at 1536, interleaved. If the 512 run solves 8 or more of 10, the default is adequate and this note overstates the problem. We expect 3 or fewer at 512 and 8 or more at 1536. We will publish the paired run, or say we did not run it, by 31 December 2026.