Image models now draw the next screen for GUI agent training; 9.3% of imagined Android rollouts survive ten steps, and neither paper counts the tap that changes nothing

Two papers in a week replace the emulator with an image model: show it a screen and an action, and it draws the screen that would follow. The gains are real. What decides how far they go is how the drawn screens are checked, and both papers check with another model.

Two papers posted between 1 and 5 October 2026 train GUI agents on screens an image model drew instead of screens a device produced, and both report gains on live benchmarks. The limit shows in the filter logs: Infinite-Dreamer's VLM checker passes about 80% of generated steps, so only 9.3% of its imagined Android rollouts are still clean at step ten, and the hard AndroidWorld tasks, the long ones, got worse with synthetic data mixed in. Neither paper reports how often a generated step is a tap that changes nothing. In phoneshell's real iPhone crawls, that was 41 of 237 taps.

What the two papers did

Infinite-Dreamer (Ning, Niu, Xing, Duan, He, Wang and Wang; arXiv:2610.05861, 5 October 2026) turns a screen transition into an image edit. A VLM reads a real before-and-after pair from AndroidControl and writes a "delta-text": the action, the elements added, the elements removed. FLUX.2 Klein (4B and 9B) is fine-tuned to draw the after-screen from the before-screen and the delta-text. An LLM "App Designer" then plans multi-step tasks, delta-texts are written from its plan, and the editor rolls them out. GLM-4.7v scores each step and drops anything under 90 out of 100. Qwen3-VL trained on this output alone, with no real trajectories, is called Infinite-Actor. The code link in the paper returned 404 on 6 October.

AutoGUIWorld (Yang, Wu, Huang, Zhang, Chen, Chen, Zhao, Deng and others; arXiv:2610.01215, 1 October 2026) does the same on desktops. It samples an operating system, a visual style and an interface state, renders a seed screen, has a planner state each atomic action with its intended visual effect, and has an image generator edit the current screenshot into the next. A Gemini checker audits every step on five criteria. After target-box repair and filtering, 79,266 step-level samples remain across Ubuntu, Windows, macOS and Chrome.

The gains, and where they stop

Infinite-Dreamer's agent results are clean at the short end. With synthetic data only, AndroidWorld Pass@1 rose from 22.56 to 31.61 at 2B, 30.17 to 39.37 at 4B and 39.37 to 43.82 at 8B; MobileWorld Pass@1 at 8B went from 8.77 to 15.20. The ablation at 4B is the more useful table: real trajectories alone reach 42.67, synthetic alone 39.37, real plus both synthetic kinds 48.28. So the generated data is an addition to real data, not a replacement for it, and each figure is one training run.

The rollouts thin out with length. Each step passes the filter about four times in five, steadily, so whole trajectories decay geometrically: half are gone by step four, nine in ten by step ten. Hard AndroidWorld tasks need long, strictly ordered sequences, and there the mixed-data 4B model scored 10.5 against 13.2 for real data only. In its 17 failed Hard episodes, 7 ran out of steps and 6 declared completion when the verifier disagreed.

AutoGUIWorld's per-action audit says which steps an image model finds hard. Clicks were inconsistent 0.67% of the time after filtering; drags 7.58%, scrolls 7.09%, typed text 6.35%. The failure examples include a terminal command replaced by a different, plausible command, and a text-entry step that jumped straight to the finished, formatted document. The macOS rate was 0.40%, Ubuntu 2.37%. The 35B model still lost on Chrome in both desktop benchmarks.

What a real device adds

Both pipelines are checked by a model judging whether a drawn screen looks like the right consequence. A device answers a different question: what did happen. phoneshell's crawler taps every control it judges safe, waits for the screen to settle, and records an edge with the before and after screen. Across the 11 crawls on the rig (Settings and Talika, 11 September 2026) there are 237 edges: 193 taps that reached a different screen, 3 that left the app, and 41 that changed nothing, meaning the accessibility text signature matched and under 1% of the screenshot moved. A Settings screen of informational rows, such as About, answers no tap at all, which is why crawl.py allows 14 dead taps in a row before it suspects a frozen phone.

Dead taps matter for a world model because agents meet them constantly and need to recognise them. Infinite-Dreamer's own failure analysis found 7 of 17 failed Hard episodes containing an action after which the screenshot was byte-identical. A generator conditioned on "elements added, elements removed" is never asked to produce an unchanged screen unless the training pairs include them, and neither paper says how many did. phoneshell's crawls are small and tap-only, so they cannot test drags or typing, the actions AutoGUIWorld found hardest. They can test the case the papers left out.

What it means if you build training data for phone agents

• Use generated screens to widen coverage of short, common transitions, and keep real trajectories for long tasks. Infinite-Dreamer's own numbers show both effects in one table.
• Report the no-op share of your training transitions, and include no-ops on purpose. A model that has only seen taps that work will read a dead tap as progress.
• Grade a world model against a device on held-out real transitions, not only with a VLM judge. A judge that scores plausibility will pass a screen that is plausible and wrong, which is the failure AutoGUIWorld documented for typed text.
• Expect action types to differ by an order of magnitude. A 2% headline inconsistency rate hides 0.67% for clicks and 7.58% for drags.

Check it yourself

• Infinite-Dreamer: Table 1 (agents), Table 3 (data-source ablation), Table 11 (survival by horizon), Appendix G.3 (Hard failures) of https://arxiv.org/abs/2610.05861. Code is announced at github.com/swaydy-n/Infinite-Dreamer; it returned HTTP 404 on 6 October 2026.
• AutoGUIWorld: Sec. 5.2 and Figures 9 to 11 (benchmarks), Appendix F.2 and Table 12 (transition audit) of https://arxiv.org/abs/2610.01215.
• phoneshell edges: python3 -c "import json,glob,collections;print(collections.Counter(e['kind'] for f in glob.glob('runtime/crawls/*/manifest.json') for e in json.load(open(f))['edges']))" prints tap 193, no-op 41, crossing 3 on the rig. The no-op rule is in phoneshell/crawl.py: snap.signature == screen.textual and moved < 0.01, plus a same-screen check for content that changes on its own.

What would prove this wrong

Our prediction is that an action-conditioned GUI world model trained on transitions that almost always change something will draw a change for most real dead taps. Test: give Infinite-Dreamer-9B, or the next released model of its kind, the before-screens and tap descriptions of phoneshell's 41 no-op edges, and score its output with the crawler's own rule. If at least 30 of the 41 come back unchanged, the models already handle dead taps and we are wrong. We will publish the run within a month of the model weights being downloadable, and say so by 31 March 2027 if they never are.