Rewriting only the code around a frozen GUI model added 10.16 held-out points on OSWorld; the 12.33 headline includes the tasks it was tuned on

Six backbones, one optimizer that rewrites the code around them, and gains of 1.49 to 10.16 points on tasks it never saw. The two numbers that matter for anyone running their own evaluation: the same optimizer fed one run per task found less than half the gain, and 12 to 20% of tasks flipped between pass and fail across three runs of an unchanged agent.

An optimizer that edits only the code around a frozen GUI model raised Qwen3-VL-32B from 43.02% to 53.18% on 201 sealed OSWorld-Verified tasks, a gain of 10.16 points (GUI-HARVEST, arXiv:2610.00948, 1 October 2026). Across six backbones the held-out gains run from 1.49 to 10.16 points. The abstract's 12.33 is measured on all 361 tasks, 160 of which the optimizer searched or selected on.

GUI-HARVEST rewrites the agent code and keeps the model fixed

The paper is by Geyi Yang, Zikun Qu, Xiang Li, Zhiyong Wang, Min Zhang, Shipei Zeng and Zhongxiang Dai (CUHK-Shenzhen, Tianjin University, Harbin Institute of Technology Shenzhen, East China Normal University, Shenzhen Research Institute of Big Data). The object it optimizes is the executable layer between model and computer: how observations are assembled, how actions run, and when the agent verifies, recovers or stops. The model weights never change.

Each round runs every search task three times, lines up what the model said with what was executed and with screenshots before and after each action, clusters failures across tasks into recurring patterns, and turns a pattern into a bounded source-code edit with its expected effect written down before evaluation. An edit is promoted only if neither the search nor the validation split declines and one strictly improves. The loop stops after 10 rounds or when all 5 candidates in a round are rejected. All four optimizer roles run on Claude Sonnet 5. Qwen and the proprietary models start from Agent S3 with a UI-TARS-1.5-7B grounder; OpenCUA starts from its own runtime.

Scores are percent at 15 steps, averaged over three runs per task. No standard deviations or confidence intervals are given for these rows. The full-suite gain is larger than the test gain for all six backbones, which is what you would expect if part of the improvement is fitting the tasks the optimizer looked at. OpenCUA-32B's +1.49 on test is about 1.2 standard errors of a single 201-task mean (our estimate, scaling the paper's 0.93 point standard error over 361 tasks at K=3), so "consistent held-out gains" holds for five of the six. Moved to WindowsAgentArena with no further optimization, GPT-5's frozen result went from 50.88% to 64.75% at 50 steps.

The optimizer only finds the gain when it sees each task run three times

The ablations are where the method's own measurement lesson sits. With one run per task as evidence (K=1), the optimizer found +4.42 test points for Qwen3-VL-32B; with three it found +10.16 (Table 3). Without screenshots in the evidence it found +1.52. The reason is in Table 8: across three runs of the unchanged starting agent, 11.8% (OpenCUA-32B) to 20.4% (Qwen3-VL-32B) of tasks scored zero in one run and positive in another. A single run cannot tell an edit that helped from a task that happened to pass. Selecting the final framework, the paper reports 43.1% on the full suite scored at K=1 against 50.9% at K=3 and 49.7% at K=5.

The edits that survived are concrete and mostly about stopping and looping. Qwen3-VL-8B's redundant action loops on search tasks fell from 42 to 27 after "loop rejection and fresh restart". That is the same direction as the Absorbed in Inertia result we covered on 1 October, where only a context reset broke repeat loops. Premature completion fell from 28 to 15 for Qwen3-VL-32B and from 20 to 10 for Gemini 3.1 Pro, through gates that withhold the first done. Infeasibility misjudgment recurred for all six backbones. On cost the edits were neutral or better for two of three models: GPT-5 went from $78.6 to $72.3 for the 361 tasks and Gemini 3.1 Pro from $140.0 to $92.4, while Qwen3-VL-32B rose from $27.3 to $34.2. The optimizer's own Claude spend is excluded and not reported.

What it means for a phone agent evaluation

On phoneshell's 9 September 2026 board, six frontier models span 89.7% to 96.6%, a 6.9 point spread, and none of the 15 pairs separates under McNemar. GUI-HARVEST's median test gain across six backbones is 5.78 points from code alone. Our judgement, across two different benchmarks: the framework is a lever about as large as the choice among frontier models, and it deserves the same evaluation discipline.

That discipline is what our board lacks. It is one run per model and task (349 rows for 6 models and 58 tasks in run xv2). At the board's pooled pass rate of 92.3%, the binomial standard error of one model's score over 58 tasks is about 3.5 points, our arithmetic. A framework change scored the same way would need to move a model by roughly 7 points before a single run could show it, more than GUI-HARVEST found for five of its six backbones. Three decisions follow for phoneshell: score any framework change at three runs per task, keep a sealed split the change was not designed on, and write the expected effect into the commit before the run, which is the part of GUI-HARVEST's method that costs nothing.

Check it yourself

• The paper: https://arxiv.org/abs/2610.00948. Table 1 (test and full per backbone), Table 4 (search and validation scores), Table 3 (ablations), Table 8 (baseline flip rates), Table 9 (failure counts before and after), Appendix G.7 (split: 80 / 80 / 201, 8 Google Drive tasks excluded).
• The code: https://github.com/GaryYang12345/GUI-HARVEST, created 29 September 2026. The README in its demo folder says the optimized frameworks are "coming soon"; the split seed is passed as --seed and its value is not in the paper.
• The in-sample share: Full counts 361 tasks, Test 201, so 160 of 361 (44%) of the full-suite tasks were visible to the search or selection.
• phoneshell's run count: wc -l runtime/bench/xv2.jsonl on a phoneshell install with local results (349 lines); the board itself is in site/results.json under board.

What would prove this wrong

If the optimized frameworks are released, an independent rerun of Qwen3-VL-32B on the 201 test tasks at K=3 and 15 steps should score at least 50.0%, which is seven points over the 43.02% starting point. We expect that to be reported by 31 March 2027. A rerun below 48.0% would mean most of the gain was specific to the authors' runs. On our side, by the same date we will score phoneshell's next framework change on one model at three runs per task over the 58 public tasks, with the predicted effect in the commit message, and report the flip rate of the unchanged framework. We predict that flip rate is between 5% and 15% of tasks.