Learned computer-use policies win on repeats, not first tries: NSPI's OSWorld Pass^1 sits below the agent it was learned from on both models

Neuro-Symbolic Policy Iteration turns one agent run into a refined script and beats the agent on Pass^3 in all four settings, at 15 to 217 times lower cost per run. On OSWorld its first-try success is 2.3 and 1.2 points lower. The gain is consistency, and its judges cannot see failures that never reach the screen.

Turning a computer-use agent's run into a refined script buys reliability across repeats, not first-try success. In Neuro-Symbolic Computer Use (arXiv:2609.36927, 29 September 2026), the learned policies win Pass^3 (solved in all three runs) in all four settings, but on OSWorld their first-try rate is below the agent they were learned from with both models: 53.6% against 55.9% on GPT-5.6 Terra, 72.9% against 74.1% on Claude Opus 5. What they remove is the drop between the first run and the third: 3.4 points on average, against 12.4 for the agent.

What Simular's researchers released on 29 September 2026

The paper is by Hyewon Suh, Thanh Minh Nguyen, Chih-Lun Lee, Darrow Hartman, Lizhao Liu, Xin Eric Wang, Ang Li and Jiachen Yang (Georgia Institute of Technology and Stanford University, work done at Simular). It proposes Neuro-Symbolic Policy Iteration (NSPI): start from one agent trajectory, convert it to a program in Simular's Simulang scripting language, replace pixel coordinates with element descriptions that a grounding model (UI-Venus-1.5-30B-A3B, consensus of three calls) resolves at run time, run it, judge it, let the agent continue from the point of failure, and have a coding model rewrite the script. The loop never sees the benchmark's evaluator until the end.

The test set is 361 OSWorld-Verified tasks (Google Drive tasks excluded) and 169 ScienceBoard tasks, each run three times, with GPT-5.6 Terra and Claude Opus 5 as the base agents. Baselines are the base agent, Batch2Code (the same script format, no refinement loop) and ASI (an agent that calls learned skills).

Break-even is the build cost divided by the per-run saving, and the table's values reproduce from its other columns (2.03 / (0.081 minus 0.0054) = 26.8). In wall-clock time the break-even is 8.6 to 16.9 runs.

The gain is in Pass^3, and on OSWorld with Opus it is 3.6 points

The abstract says the policies beat the base agent's Pass^3 by 3.6 to 15.8 points, which matches Table 1; the introduction says 4.5 to 13.9, which does not, so use the table. The spread matters. On ScienceBoard with GPT-5.6 Terra, where the agent solves only 13.6% of tasks three times out of three, the script nearly doubles it to 26.0%. On OSWorld with Claude Opus 5, the agent is already consistent (74.1% first try, 68.4% all three), and the script adds 3.6 points of Pass^3 while losing 1.2 of Pass^1. There the case for the script is cost: $0.0097 a run against $2.10, repaid after about three runs.

Batch2Code separates the two effects. It uses the same script format with no refinement, and its curve is just as flat (OSWorld with Opus: 46.7 to 45.2), but it sits 3.3 to 33.1 points below NSPI on Pass^3. A script makes behaviour repeatable. Only the judge-and-heal loop makes it repeatably right. Without the loop, Batch2Code's OSWorld Pass^1 on Opus is 27.4 points below the agent's.

Two results limit the method. On OSWorld-V2, where tasks take skilled humans over an hour, agent trajectories averaged 158 steps and the converted scripts averaged 970 lines, and NSPI trailed the agent by 5.6 points of strict success; the authors attribute it to scripts recomputing values the agent had read off the screen. And the reuse test is narrower than the name suggests: policies refined on generated variants reached 55.0% Pass^3 on held-out parameter changes (AutoRPA 37.5%) and 60.0% on changed starting states (AutoRPA 51.4%), over 40 and 35 task families.

Two checks guard the loop, and both have measured blind spots

The judges are conservative by design, and that costs money rather than reliability: rejecting good policies means more healing rounds. The step judge's blind spot is sharper. On 19 OSWorld policies annotated by people, it located the first wrong step in 9 of 10 cases where the failure was visible on screen and 0 of 6 where it was not. A judge that only watches the screen cannot localize a wrong value written to a file.

The deployment verifier shows the usual trade-off with numbers attached. The model that rarely blocks good runs (Qwen3.6-35B-A3B, at most 8.4% wrong blocks) let 17 of the 24 steps that should have been stopped go through. The one that caught more (Qwen3.8-27B, 6 of 24 passed) wrongly blocked up to a quarter of correct runs. Twenty-four positives is a small test, and the paper says so.

What it means for a phone agent that replays macros

phoneshell already has the symbolic half. phoneshell/agent/macros.py stores a recorded sequence of steps, each with an optional anchor (text that must be on screen before the step fires) and an optional expected text afterwards; replay stops and hands the screen back to the model when an anchor is missing. Three gaps separate it from NSPI. It has no refinement loop: a macro is saved once and never rewritten from its own failures. It does not check outcomes: the expect field exists, but none of the seven recorder.record calls in phoneshell/mcp_server.py passes one. And its claimed benefit is unmeasured: the module docstring says replay is "worth roughly twenty points of task success on repeat work", and no run in FINDINGS.md backs that number.

The paper also says what to measure. phoneshell's largest sweep (9 September 2026, our local results file) ran each of 349 task and model pairs exactly once, so it reports Pass^1 only. NSPI's main effect would be invisible in that design: on OSWorld the scripts lose a point or two of Pass^1 and gain 3.6 to 15.8 points of Pass^3. A macro evaluation on the phone needs at least three runs per task, from a reset state, with the macro and without it.

Our judgement on cost: on a physical iPhone the per-run saving is real but smaller than on a desktop VM, because phoneshell's wall clock is dominated by device calls (a tap is 550 ms on a light screen) rather than model calls. The dollar break-even of 3.1 to 26.8 runs would likely hold; the 3.4 to 5.1 times speed-up would not, and we have not measured it.

Check it yourself

• The paper: https://arxiv.org/abs/2609.36927; Table 1 (Pass^1, Pass^3, cost multiples), Table 2 (variants), Table 3 (break-even), Section 4.5 (OSWorld-V2), Section 4.6 (verifier), Tables 5 and 6 (judges).
• Recompute the break-even: python3 -c "print([round(c/(a-p),1) for c,a,p in [(2.03,.081,.0054),(5.68,.52,.0079),(3.09,.48,.0072),(6.45,2.1,.0097)]])" gives 26.8, 11.2, 6.5, 3.1.
• Recompute the average Pass^1 to Pass^3 drop from Table 1: agent (13.4 + 9.1 + 21.3 + 5.7) / 4 = 12.4; NSPI (3.2 + 6.1 + 3.2 + 0.9) / 4 = 3.35.
• phoneshell: grep -n "recorder.record" phoneshell/mcp_server.py shows seven calls and no expect=; sed -n 1,15p phoneshell/agent/macros.py shows the twenty-point claim; grep -n "twenty" FINDINGS.md shows no matching measurement.

What would prove this wrong

By 31 March 2027 we will run 20 phoneshell tasks three times each from a reset state, once with a saved macro available and once without, on the same model. If macros raise Pass^1 by 10 points or more, the claim here that compiled routes mainly buy consistency rather than first-try success does not carry over to the phone. We expect Pass^1 to move by less than 5 points and Pass^3 by more than Pass^1. Separately, if a later version of the paper reports OSWorld Pass^1 for NSPI above the base agent with both models, the headline of this note no longer holds for that version.