Desktop-only training data lifted Qwen3.5-4B's mobile grounding by 10.8 points; phoneshell's crawler keeps centre points that cannot score a click

The dataset is desktop-only, built from apps chosen because their accessibility trees are good. The gain still shows up on phone screenshots. We compared what DeskForge stores for each observation with what phoneshell's crawler stores for each iPhone screen.

Fine-tuning Qwen3.5-4B on 200,000 grounding examples drawn from Linux desktops raised its ScreenSpot-v2 mobile accuracy from 82.44% to 93.21%, a 10.77-point gain from a dataset that, in its authors' words, "contains no mobile screens". The same model, used as the action model under a fixed planner, solved 50 of 119 WebArena-Infinity tasks instead of 31. DeskForge was posted on 1 October 2026. phoneshell's crawler collects the same kind of data on an iPhone, and stores it in a form that could not have produced either number.

What DeskForge built

A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong, Marc Pollefeys and Peter W. J. Staar (ETH Zurich, IBM Research Zurich, Microsoft; arXiv:2610.02320) compose real applications into controlled desktop scenes: 19 Linux apps from GNOME Calculator to VS Code, 7 appearance presets including four macOS-inspired and one Windows-inspired theme, all on a Linux backend, and 7 resolutions from 1366 x 768 to 3840 x 2160. They vary app state, staged content, window layout and stacking. Each observation fuses the screenshot, the accessibility tree and window geometry: boxes are clipped against ancestors, the viewport, higher windows and overlays, visible area is stored as a union of rectangles, and displayed text is separated from accessibility names so an icon's name is not taken for on-screen text. Each click records what happened to every element: appeared, disappeared, moved, text or state changed, newly occluded, revealed.

The mobile gain, and who it went to

Four vision-language models were LoRA-tuned (rank 8, one epoch) on 200,000 examples and tested on five external grounding benchmarks; all four improved on all five. Qwen3.5-4B gained 11.51 points on ScreenSpot-Pro (30.17 to 41.68) and 10.11 on OSWorld-G (51.24 to 61.35). The per-domain ScreenSpot-v2 split, in the paper's Table 11, is where phones come in. Qwen's mobile score rose about as much as its web score and less than its desktop score. UI-R1-3B, a model already trained for GUI grounding, gained 0.6 on mobile. Judgement: the transfer is real for a general model with weak GUI grounding and close to nothing for one that already has it.

The long-horizon test changes only the action model under a fixed Qwen3.6-27B planner, with success decided by each environment's own state checks. On a custom 119-task WebArena-Infinity panel Gemma4-E4B went from 5 to 40 solved, Qwen3.5-4B from 31 to 50, InternVL3.5-8B and UI-R1-3B each gained 7. On the 100-task OpenApps longer-horizon set Qwen went from 3 to 15. The fine-tuning used single grounding examples, not trajectories.

Why phoneshell's crawl data cannot do this yet

phoneshell's crawl.py walks an app, taps what it judges safe, and writes a manifest.json per run. The 11 manifests on the rig, all from 11 September 2026, hold 113 screens from Settings and Talika. For each control they keep the label, the accessibility type, a key, the centre point and a skip reason. They do not keep width, height or a box, so a model's predicted click cannot be scored as inside or outside the element, which is how ScreenSpot-style benchmarks grade. Each edge records the screen a tap left, the screen it reached and whether it was a tap or a no-op. Nothing records what changed on the screen. The crawler's caution shows in the skips: 54 switches, skipped because a switch changes a setting, and 29 "dictate" controls were never tapped, so state changes are exactly what the data lacks.

The grounding question matters to phoneshell even though its agent taps element ids. When the accessibility tree is thin, perception/blind.py recovers elements from pixels with OCR and an icon proposer, and those elements are only as good as their boxes. DeskForge's admission rule is a warning for anyone copying its recipe to phones: apps whose trees did not expose content were rejected before collection, so its 99.84% element correctness describes apps chosen for good trees. The apps where a phone agent most needs pixel grounding are the ones that rule removes.

What it means if you collect phone GUI data

• Store the box, not the centre. The centre is derivable from the box; the reverse is not, and every grounding benchmark scores against the box.
• Record per-element effects on each action, the six DeskForge categories at minimum. A from-to screen pair cannot say whether a toggle flipped.
• Report how many captures your audits reject and why. DeskForge's 4.7% is the only published reference rate we know of; it gives no breakdown by cause.
• Do not assume desktop data is a substitute for phone data on a model already trained for GUIs. UI-R1's 0.6-point mobile gain is the counterexample in the same table.

Check it yourself

• Domain split: Table 11 of https://arxiv.org/abs/2610.02320; long-horizon: Table 4 and Sec. 5; dataset statistics: App. A; audits: App. B.3 and Table 10. Code, data and model: https://saidgurbuz.github.io/deskforge/.
• phoneshell crawl fields: open any runtime/crawls/<run>/manifest.json and list the keys of screens[0].controls[0]: label, type, key, x, y, skip.
• Totals: python3 -c "import json,glob;m=[json.load(open(f)) for f in glob.glob('runtime/crawls/*/manifest.json')];print(len(m),sum(x['summary']['screens'] for x in m),sum(len(s['controls']) for x in m for s in x['screens']),sum(x['summary']['edges'] for x in m))" prints 11 113 1922 237 on the rig.

What would prove this wrong

Our reading is that DeskForge's mobile gain is transfer to a weak GUI model, not evidence that desktop data substitutes for phone data. Test: score the released DeskForge-tuned Qwen3.5-4B and its base on phoneshell's 113 crawled iPhone screens, after adding element boxes, on point-in-box accuracy for every labelled control. If the tuned model gains 8 points or more there, desktop data transfers to real iPhone screens as well as it did on ScreenSpot-v2 and our caution is wrong. We expect a gain under 5 points. We will publish the run, or say we did not run it, by 31 March 2027.