Local GUI agents fail mostly by stopping early, and a stop check alone is worth 2.9 points: LocalLSTC on OSWorld

With a 9B local planner, 78.6% of failed desktop-agent runs end in a stop without evidence of completion. LocalLSTC's final check rejects a stop in 32.1% of tasks, yet removing it costs less than two standard deviations; the kept subgoal is what makes a rejected stop recoverable.

When four desktop-agent frameworks swap GPT-5 for a local Qwen3.5-9B planner, OSWorld success falls from 60.9% to 35.2%, and 78.6% of the failed local runs end with the agent stopping without evidence the task was done. LocalLSTC (arXiv:2608.25777, revised 25 September 2026) adds a final check that rejected a stop in 32.1% of tasks. Removing that check costs 2.9 points, well within two of the system's own 2.7-point standard deviations. Removing the kept subgoal costs 9.6.

What the revision reports

LocalLSTC is by Weiming Li, Helen Paik and Yulei Sui (The University of New South Wales). Version 1 appeared on 26 August 2026 and version 2 on 25 September. It is a training-free control layer for GUI agents that run on local models. It keeps a persistent record of the current subgoal, the evidence gathered so far and runtime feedback, and routes each step as continue, advance, stall, error or finish. Two mechanisms carry the design: long-to-short planning (L2S) forms each commitment from that record, and short-to-long control (S2L) writes execution results back into it. S2L includes a Final Verification step that can refuse a finish.

All runs are on OSWorld-Verified (369 Linux tasks) and WindowsAgentArena (154 Windows tasks), on one workstation with a 96 GB RTX Pro 6000 Blackwell, grounding by GTA1-7B. The failure labels are assigned by GPT-5.6-sol and checked against two human annotators on 185 trajectories.

The GPT-5 figures are the frameworks' reported leaderboard results, not reruns on the authors' machine, so the 25.7-point drop mixes a backbone change with a change of who ran it.

The commonest failure is a premature stop, and the stop check barely pays

The paper's annotation guide defines Stop as "termination without sufficient evidence that the task has been completed", where required conditions remain unverified or unsatisfied and execution does not resume. Under a 9B planner it is the leading failure by a wide margin: 78.6% of failed trajectories, against 42.2% for failed recovery. Two human annotators agree on the Stop label at kappa 0.665, in the same range as the other four labels (0.615 to 0.792), so the exact rate is soft. The ranking is not.

A check aimed at that failure should be the biggest lever. It is not. Final Verification fires, rejecting a finish in 32.1% of tasks, and those tasks end up scoring 38.9%. But the ablation without it scores 44.7% against the full system's 47.6%, a 2.9-point difference. That is about 11 of 369 tasks, and the full system's own standard deviation over two runs is 2.7 points. Stall and loop handling (minus 2.5) and state-conditioned routing (minus 3.7) are in the same range. Four removals clear two standard deviations: L2S (minus 11.2), the persistent-subgoal interface (minus 9.6), candidate proposals (minus 6.9) and S2L as a whole (minus 6.1).

Table 4 explains why. A rejected stop only helps if the agent has something to go back to. With the full system, tasks that hit a rejected termination score 38.9%. Without L2S, which is what builds commitments from the kept subgoal, the same event is rarer (19.3% of tasks) and those tasks score 23.1%. Refusing a false "done" hands the problem back to a planner. What that planner still knows about the task decides whether the second attempt goes anywhere.

What it means for a phone agent that stops itself

phoneshell's loop has no Final Verification. When the model calls phone_done the loop breaks and the device checks run once, as the grade, never as feedback (phoneshell/bench/openrouter.py). That separation is deliberate: the checks are the benchmark's ground truth, and letting the agent see them would teach to the test. But it means every premature stop is final. Our 15 September note counted 16 of 27 device-graded failures in the 9 September sweep ending with the model saying the job was done. In LocalLSTC's terms, that is the Stop label, measured on a phone by the phone instead of by an annotator.

The LocalLSTC numbers argue against the cheap fix, which is a model-side "are you sure" pass before phone_done is accepted. On OSWorld such a pass fires often and recovers little unless the agent has kept an explicit subgoal to resume from. It also has a price: running without Final Verification cut tokens per task from 252.2k to 227.8k, so the check costs about 24k tokens a task for its 2.9 points. For a phone agent the order of work is state first, then a verifier that reads the device rather than the model's own transcript. Our judgement: a verifier that re-reads the accessibility tree for the requested end state is closer to what phoneshell's checks already do than to LocalLSTC's model-judged verification, and it would be cheaper.

The scale result is the other takeaway for anyone running agents locally. With the same control layer, OSWorld SR-100 goes from 28.1% on Gemma-4-E4B and 37.3% on Qwen3.5-4B to 47.6% on Qwen3.5-9B, 60.3% on Gemma-4-26B-A4B, 64.7% on Qwen3.6-27B and 73.7% on Qwen3.8-27B. The 27B result is above the 72.1% the paper lists for Claude Sonnet 4.6. None of these are phone-sized models; the smallest that clears 45% here needs 9B parameters and a 96 GB GPU in the authors' setup.

Check it yourself

• Paper: https://arxiv.org/abs/2608.25777 (v2, 25 September 2026); Table 1 (backbones, with SD), Table 3 (ablations), Table 4 (rejected termination), Tables 6 to 8 (annotation agreement and label incidence), Appendix C.2 for the Stop definition.
• Noise test: full 47.6 with SD 2.7 over two runs; any ablation above 42.2 is within two SDs. 2.9% of 369 tasks is 10.7.
• phoneshell: grep -n "if tools.done is not None" -B 2 phoneshell/bench/openrouter.py; grep -n "run_all(self.phone" phoneshell/bench/runner.py.

What would prove this wrong

Dated 31 March 2027. If LocalLSTC or a replication reports the Final Verification ablation over at least three runs on OSWorld with Qwen3.5-9B, the gap to the full system will be under 4 points. A gap of 5 points or more, with an interval that excludes zero, would mean the stop check matters on its own and our reading of Table 3 was too quick. For phoneshell: if we add a device-reading completion check that returns a rejected phone_done to the model once, it will convert fewer than half of the claimed-done failures on our 80 tasks into device passes.