Phone agents learn where buttons are from fine-tuning and how tasks go from the prompt: splitting experience beat both on MobileGym and AndroidWorld

Three small mobile agents, two Android-style environments, one pool of their own experience split four ways. Where a button sits belongs in the weights; what a task usually does next, and what must be true before an action works, belong in the prompt. A prompt-only phone agent such as phoneshell can only use half of that, and currently stores less than half.

Where a button sits is learned better by fine-tuning; what a task usually does next is used better from the prompt. Pooled over three 7 to 8 billion parameter backbones on MobileGym, element locators from the agents' own trajectories gained 6.7 points of success when trained into the weights and 1.8 when written into the prompt, while recurring procedures gained 5.2 in the prompt and 1.0 in the weights (Wu, Ding and Huang, arXiv:2610.01787, 1 October 2026). Sending each kind to its better destination beat putting everything in either one.

The paper splits a trajectory into four kinds of experience

Beining Wu, Zihao Ding and Jun Huang (South Dakota State University) start from a disagreement in the self-improvement literature: some studies find that fine-tuning on an agent's own trajectories beats retrieving them into the prompt, others find the reverse. Their explanation is that a trajectory bundles items with different properties, so the answer depends on the mix. They split the pool into locators (a named element and where it sits), procedures (a run of three action targets that recurs within a task family), state facts (a precondition that must hold before an action succeeds, taken from a failed and a successful attempt at the same target) and lessons (an action that failed in a state, paired with the one that later worked there). On MobileGym that is 9,907 locators, 51,052 procedures, 3,980 state facts and 24,651 lessons.

The environments are MobileGym, a simulated phone with 20 Chinese-language apps and 416 task templates, and AndroidWorld, 116 tasks on 20 apps in an emulator, with apps held out for testing in both. The backbones are GUI-Owl-1.5-8B, ScaleCUA-7B and Qwen3-VL-8B, three seeds each, with paired bootstrap intervals. Training is LoRA rank 16, with preference pairs for lessons and supervised fine-tuning for the rest. Code and data are not yet released.

Against the better single destination for each backbone, routing adds 5.2, 2.5 and 5.6 points on MobileGym and minus 0.3, 5.0 and 3.2 on AndroidWorld, our subtraction from Table 2; the mean is the abstract's +3.5. The one cell where routing does not help, GUI-Owl on AndroidWorld, has an interval of minus 2.6 to plus 2.0. Routing the wrong way round is worse than either single destination in every cell, which is the strongest sign that the split is real rather than a tuning artefact.

Two numbers measured before training predict the destination

The rule uses two properties of each component, both measured on the pool before any training. Recurrence is the share of unrelated tasks in the same app whose episodes contain the item. State-conditionality is how much the item's content changes when the screen state changes. The fitted score is 11.29 times recurrence minus 5.73 times state-conditionality minus 1.28; positive goes to the weights. Fitted on two backbone families and tested on the third, it recovered the destination in 24 of 24 cells, 19 of them with intervals clear of zero. Lessons are the soft spot: the abstract says they win in the weights, but 3 of their 6 cells have intervals that include zero.

Training also changes what the prompt is worth afterwards. Before fine-tuning, adding a note changed the agent's action on 42% of probed steps. After the same items were trained in, the rate fell to 16% for locators, 15% for lessons and 22% for procedures, and it fell most for the items that recur most (Spearman minus 0.92). The prompt route is cheap: 14.2k against 13.2k prompt tokens per episode, a ratio of 1.07. The weights route for everything cost 15.4 H100 hours; the routed version 6.1.

What it means for a prompt-only phone agent

phoneshell drives API models, so the weights route does not exist for it. Everything phoneshell/memory.py learns goes to the prompt through hint(): how many times the agent has been on this screen, up to eight lines of the form "control led to that screen, N times", and one free-text note per screen. The stored record (ScreenMemory) holds a layout fingerprint, controls with their outcomes, and the screen's measured settle time.

Mapped onto the paper, that is a one-step version of procedures, no state facts and no lessons. Locators are mostly moot for us: the accessibility tree gives every element an index on every step, and the docstring says a remembered control is resolved against the live tree before it is tapped. Our judgement is that the paper's largest weights win, locators, is the component a tree-reading agent needs least, because the paper's agents act by coordinates on screenshots. The component the paper finds best served by the prompt and that phoneshell does not store at all is state facts, which gained 4.6 to 5.4 points in the prompt across the three backbones on MobileGym (Table 1). A state fact is cheap to capture from data phoneshell already has: the same target tried twice on one screen, failed then succeeded, with whatever differed between the two attempts.

Two caveats limit how far this travels. The environments are Android-style and one is simulated, the backbones are 8 billion parameters or smaller, and the absolute success rates are 17 to 35%; a frontier API model on a real iPhone starts far higher, and phoneshell's board sits at 89.7 to 96.6%. A prompt gain measured from a base of 20% may not survive a base of 90%. And the information gap result in Section 6.1 says prompt notes help most when the producer knew something the consumer did not (context gain rising from +0.2 to +11.3 across sixths of the gap), which favours notes written by a stronger model or a human over notes an agent writes for itself.

Check it yourself

• The paper: https://arxiv.org/abs/2610.01787. Table 2 (held-out success by arm), Table 1 (prompt and weights gains for procedures and state facts), Table D.2 (route gain per cell with intervals), Section 5.1 and Appendix D (the rule and its 24 cells), Table E.4 (readout after training), Table H.1 (cost).
• Routing against the better single arm: subtract max(all-in-prompt, all-in-weights) from routed in each column of Table 2.
• phoneshell's memory: grep -n "class ScreenMemory" -A12 phoneshell/memory.py and grep -n "def hint" -A28 phoneshell/memory.py; nothing in the file records a precondition or a failed-then-succeeded pair.

What would prove this wrong

Two predictions, dated 31 March 2027. When the code is released, a replication with a fourth backbone family on AndroidWorld will put locators in the weights and procedures in the prompt (the two large effects, route gains of +4.9 and minus 4.2), while lessons will come out on the prompt side in at least one of three seeds. And on phoneshell: we will add state-fact notes (target, screen, what differed between the failed and the successful attempt) and run the 58 public tasks three times with and without them on one model. Because the board starts above 89%, we predict the pass rate moves by less than 3 points, and turns per task fall by at least 10% on tasks where a state fact fired. If turns do not fall, prompt-side state facts do not carry over from a 20% base to a 90% one.