A 'screen did not change' notice barely stops computer-use agents repeating themselves; wiping the history changes the next action 96 to 100% of the time
Across four open agents, a factual notice about a no-effect action moved repeats by minus 23.4% to plus 16.9%, and deleting just the duplicate attempts still left 61 to 74% of next actions unchanged. Only a full context reset broke the loop. phoneshell's live loop defence is the notice; its loop detector is never called.
Telling a computer-use agent that its last action left the screen unchanged does little to stop it repeating that action. Across four open agents on 97 OpenComputer tasks, such a notice moved the rate of repeated no-effect actions by between minus 23.4% and plus 16.9%, and none of the four changes exceeds two standard errors (arXiv:2609.37176, 29 September 2026). Wiping the conversation history instead changed the very next action 96.4 to 100% of the time. phoneshell's only live defence against loops is that notice.
What Neuchâtel and Delft measured
"Absorbed in Inertia: Activation Analysis for Computer-Use Agents" is by Giulio Segalini, Zhi Wen Soi, Jérémie Decouchant and Lydia Chen (Université de Neuchâtel and Delft University of Technology). It defines inertia as two things at once: an exact screen and action pair that recurs and leaves the screen unchanged, detected mechanically, and reasoning text that admits the approach is failing, judged by a separate model that sees only the task and the reasoning. The agents are UI-TARS-1.5-7B, GUI-Owl-1.5-8B-Think, OpenCUA-32B and EvoCUA-32B-S2, on OSWorld and OpenComputer.
Recurrence appears in up to 66% of failed trajectories, more than in successful ones. For three of the four models, 55 to 89% of recurrences come right after reasoning that acknowledges the failure; EvoCUA acknowledges least. The paper's example is UI-TARS writing that Ctrl+A "didn't work", then pressing Ctrl+A again on the same screen. A linear readout of the model's activations just before the action separates recurrent from other actions with held-out AUC 0.72 to 0.95.
The interventions, all compared in paired runs against the unmodified agent: a failure notice (after two unchanged repeats, a factual line added to the instruction saying the action was performed on this screen N times and the screen did not change, history kept); activation steering away from the recurrence direction; Permanent Reset (drop all history, continue from task and current screen); and R3 (reset, try up to three actions until one changes the screen, then restore the earlier history with that action appended).
| Intervention | Change in repeated no-effect actions (UI-TARS / GUI-Owl / OpenCUA / EvoCUA) | Change in task score, points | Source |
|---|---|---|---|
| Failure notice | -4.6% / -5.0% / -23.4% / +16.9% | +1.00 / +0.20 / +2.20 / -3.60 | Table 1 |
| Activation steering | -5.6% / -0.2% / +39.0% / -0.9% | +2.20 / -2.80 / +4.20 / -0.60 | Table 1 |
| Permanent Reset | -14.6% / -28.8% / -2.3% / +6.9% | +0.40 / +3.20 / -0.50 / -4.60 | Table 1 |
| R3 (reset, reroute, restore) | -16.9% / -38.0% / -42.7% / -33.8% | +0.60 / +1.20 / -2.50 / -0.90 | Table 1 |
Changes in repeats are relative changes in the per-action rate; task score changes are in points. Task-level standard errors for the notice row are 8.4, 16.6, 16.9 and 17.6 on repeats, so none of the four changes is more than 1.4 standard errors from zero. The paper reports R3's reductions with confidence intervals below zero for all four models; no other arm reduces repeats for all four.
Deleting the duplicates is not enough; the whole history has to go
Table 2 replays the exact states where an agent had just repeated a no-effect action and asks each variant for its next action, without executing it. Unmodified, the agents propose the same action again 79.8 to 87.5% of the time. Steering along the direction that the activation readout found still yields 68.5 to 85.7%. Selective reset, which deletes only the duplicate attempts, keeps everything else and names the repeated action in the prompt, still yields 60.7 to 73.9%. A full reset yields 0.0 to 3.6%.
Read together, the notice and the selective reset say the same thing: information about the failure, in the prompt or in the activations, does not by itself move the action. The earlier context does. The authors' reading is an "absorbing region" in activation space; ours is narrower and needs no internals: if the transcript leading up to the stuck state stays in the context, the model tends to reproduce the action that transcript leads to.
Breaking the loop is not the same as finishing the task. R3's task-score changes are plus 0.60, plus 1.20, minus 2.50 and minus 0.90 points, none distinguishable from zero. Part of R3's reduction comes from stopping: when three reroute attempts all fail to change the screen, the run ends, which happened in 19.59% of trajectories. And the reset thresholds were picked on the same OpenComputer comparisons they are reported on, with no held-out selection set, which the paper states.
What it means for a phone agent
phoneshell already detects the condition the paper measures. After every action, _act_and_report in phoneshell/mcp_server.py compares the screen signature with the one before; if they match and no alert is up, the tool replies "THE SCREEN DID NOT CHANGE" and tells the model to try something different. Both system prompts (rule 4 in mcp_server.py, and SYSTEM in phoneshell/bench/openrouter.py) say the same. That is the failure notice arm, firing after one unchanged action instead of two, with the whole history kept. On the paper's numbers it is the weakest of the four interventions at reducing repeats.
The stronger mechanism exists in the repo and is switched off. StuckDetector in phoneshell/agent/verify.py tracks the last eight screen signatures and actions and diagnoses three identical unchanged repeats or an A-B-A-B-A-B ping-pong. Its docstring says loops are the failure class models cannot fix on their own. Nothing calls it. Even wired in, it only produces a diagnosis string, which would be one more notice unless the caller acts on it by trimming the context.
How often our agents repeat, we can only bound. phoneshell's local audit log (runtime/logs/audit.jsonl, 7 to 30 September 2026, not in the public repo) records 426 tap and type actions without screen signatures. Thirteen repeat the previous action on the same element within 60 seconds (median gap 9.5 seconds): ten pairs and one run of four, eight of the thirteen on a "General" button at (220, 950). Without signatures we cannot tell which left the screen unchanged, so 3.1% is an upper bound on exact recurrence in that log, not an inertia rate.
The decision for anyone building a phone agent: if loops matter, do not stop at a stronger warning. Count unchanged repeats per screen and action, and at a small threshold call the model with only the task and the current screen, accept the first action that changes the screen, then splice it back into the history. Log every reset; the paper's 19.59% early terminations are a reminder that a loop breaker can also end runs that a retry would have saved.
Check it yourself
- The paper:
https://arxiv.org/abs/2609.37176; Table 1 (paired changes with standard errors), Table 2 (same action again), Section 4.2 and Figure 7 (screen changes and returns), Appendix D (notice template, thresholds, three-attempt cap). The abstract links an anonymous code release (anonymous.4open.science/r/vlm-agent-defense-D076), which returned HTTP 401 to us on 1 October 2026, so we could not inspect it. - phoneshell's notice:
grep -n "THE SCREEN DID NOT CHANGE" phoneshell/mcp_server.py; the prompts:grep -n "does not change" phoneshell/bench/openrouter.py phoneshell/bench/runner.py. - The unused detector:
grep -rn "StuckDetector" phoneshell scripts testsreturns only the class definition inphoneshell/agent/verify.py. - The repeat count, on your own phoneshell install: read
runtime/logs/audit.jsonl, keeptapandtypeevents, and count consecutive events with the sameelement(or the sameintoandtext) less than 60 seconds apart.
What would prove this wrong
By 31 March 2027 we will wire a reset-and-reroute step into phoneshell (trigger: the same element tapped twice with an unchanged signature), add screen signatures to the audit log, and run the same 80 tasks with and without it on one model. We predict unchanged repeats per action fall by at least 30% with the reset, while the pass rate moves by less than 3 points either way. If the existing notice alone already keeps unchanged repeats under 1% of actions once signatures are logged, phoneshell's loops are too rare for this to matter, and this note overstates the risk for a tree-driven phone agent.
Sources
- Segalini, Soi, Decouchant, Chen (Universite de Neuchatel, Delft University of Technology), Absorbed in Inertia: Activation Analysis for Computer-Use Agents, arXiv:2609.37176, 29 September 2026
- HTML edition: Table 1 (paired intervention effects with standard errors), Table 2 (same action again), Section 4.2, Appendix D (notice template, thresholds)
- Blolabel, 16 of 27 failures on a physical iPhone were reported as successes by the model that failed them
- phoneshell source: phoneshell/device.py, config.py, actions.py, mcp_server.py, agent/macros.py, agent/verify.py, bench/verify.py, environments/*.yaml, FINDINGS.md (read 1 October 2026)