GroundGUI needed 586,000 screenshots and a two-stage training pipeline to reach 95.2% on ScreenSpot-V2; phoneshell's fallback icon matcher needs zero training images and was never a model

GroundGUI, posted to arXiv on 13 October 2026, pairs a new 586,000-screenshot grounding dataset with a training method called distillation-to-reinforcement learning to teach an 8B model where to tap on a screen. phoneshell's own fallback for screens with no accessibility tree is a zero-training template correlator; its primary path never needs to point at pixels at all.

GroundGUI (arXiv:2610.19100), submitted 13 October 2026 by Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao and Ritankar Das, targets GUI grounding: given a screenshot and a natural-language description of a control ("the share icon", "the field labelled Email"), return its location. The paper's starting premise is that high-fidelity training data for this exact problem is scarce across desktop, web, and mobile, despite how many benchmarks now measure it. Their fix is two contributions bundled together: ScreenRef, a dataset of 586,000 screenshots paired with 633,000-plus referring-expression instances covering text recognition, visual matching, spatial understanding, and refusal (recognising when no matching element exists), and Distillation-to-Reinforcement Learning (D2RL), a training method meant to fix two separate known weaknesses at once, poor generalisation from supervised fine-tuning alone and sparse reward signal in reinforcement learning alone. Trained sequentially with SFT then D2RL on top of Qwen3-VL-8B-Instruct, the resulting model, GroundGUI-8B, is reported at 95.2% on ScreenSpot-V2.

What it cost to reach 95.2%, and what that bought on harder benchmarks

GroundGUI-8B's reported scores vary sharply by how far the target benchmark sits from ScreenSpot-V2's conditions. On ScreenSpot-V2 itself, a comparatively easy, well-covered grounding benchmark, it reaches 95.2%. On ScreenSpot-Pro, a harder, higher-resolution professional-software benchmark, it falls to 63.9%. On OSWorld-G, a grounding-specific slice of the OSWorld desktop benchmark, 65.4%. On MMBench-GUI's L2 tier, 86.8%. On UI-Vision, the paper's hardest reported benchmark, 38.3%. The spread, from 95.2% down to 38.3% across five benchmarks for the same trained model, is itself informative: a grounding model's score is a property of the specific benchmark's difficulty and visual domain, not a single fixed capability number, which is exactly the scarcity-of-coverage problem the paper opens by describing.

phoneshell's primary path skips the problem GroundGUI exists to solve

phoneshell's (github.com/Blomega/phoneshell) default grounding path, phoneshell/perception/tree.py, never looks at pixels. It reads elements straight from the WebDriverAgent accessibility tree: type, label, identifier, and on-screen coordinates, already resolved by the operating system before phoneshell sees them. A tap targets an element by that identity, not by a learned prediction of where a glyph probably is. This is the same design fact an earlier survey on this site established directly: across thirteen real iPhone screens, every word macOS Vision OCR could read was already present in the accessibility tree on eleven of them (render-gap-ocr-vs-accessibility-tree), with Maps and Books, where labels are drawn into map tiles or printed on cover art, the only exceptions.

GroundGUI's entire contribution, a 586,000-screenshot dataset and a new two-stage training method, exists to solve a problem phoneshell's primary path never encounters: deciding where a control is from pixels alone, with no tree to read. phoneshell only reaches for anything resembling that problem through phoneshell/perception/icons.py, used "when the tree gives us nothing," per the module's own docstring. Its method is not a trained model at all: a small set of rendered glyph templates (a heart for like, a paper plane for send, an x for close) are matched against the live screen with normalised cross-correlation, polarity ignored so a white heart on a dark background and a black heart on a white one score the same. It needs no training images, no GPU, and no dataset, because it is not learning a general visual grounding capability, only matching a handful of known shapes that happen to be common contracts between an app and its user. The module states its own limit directly: "This is a candidate generator, never an oracle. Every hit carries its score and is labelled as a guess."

What it means for anyone choosing a grounding strategy

• GroundGUI's 586K-screenshot dataset and two-stage training pipeline are the right scale of investment for a platform where the accessibility tree is unreliable or absent, which is closer to the reality on many Android apps and most desktop software than on first-party iOS apps. The problem it solves does not disappear on iOS; it mostly disappears for the eleven of thirteen first-party screen types this site has already measured as fully covered by the tree.
• A trained grounding model's score is not one number. GroundGUI-8B's own results span 38.3% to 95.2% across five benchmarks, so a single headline figure from any grounding paper, including this one, should be read as a result on a specific benchmark's distribution, not a general capability claim.
• Zero-training template matching is a narrower tool than a learned grounding model by design: it only finds shapes it already has a template for, and it says so explicitly in its own output. For an icon vocabulary that is small and stable (common first-party action glyphs), that narrowness is a reasonable trade against GroundGUI's cost in data and compute. For arbitrary referring expressions in unfamiliar third-party apps, the tradeoff would likely reverse.

Check it yourself

# phoneshell's primary grounding path never touches pixels
grep -n "class Element\|identifier\|label" phoneshell/perception/tree.py | head -10

# the fallback is explicitly zero-training and explicitly a guess
sed -n '1,20p' phoneshell/perception/icons.py

# GroundGUI's own dataset size and per-benchmark spread
curl -s https://arxiv.org/abs/2610.19100 | grep -i "screenref\|screenspot"
# full paper: https://arxiv.org/pdf/2610.19100

# the earlier tree-vs-OCR survey this article's claim about iOS screens relies on
# https://blolabel.ai/blog/render-gap-ocr-vs-accessibility-tree/

What would prove this wrong

We claim phoneshell's primary grounding path never requires a trained visual grounding model because the accessibility tree already supplies element identity and position on the vast majority of first-party iOS screens. That is wrong the moment a widely-used iOS app is found whose tree is empty or unreliable on screens outside the Maps/Books-style raster-text exception already documented, at a rate that would require something like GroundGUI's approach rather than the existing OCR and icon fallbacks. We also claim a trained grounding model's reported score varies by tens of points depending on the target benchmark, based on GroundGUI's own five-benchmark spread. That pattern is falsified if a future grounding model is shown to hold within a few points of its best score across benchmarks of clearly different difficulty. We will treat either event, checked by 31 March 2027, as grounds to revisit this comparison.

Sources

• Xie, Tian, Tang, Li, Ciobanu, Singh, Mao, Das, GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning, arXiv:2610.19100, submitted 13 October 2026, read 16 October 2026.
• Blolabel, phoneshell: phoneshell/perception/tree.py, phoneshell/perception/icons.py, read 16 October 2026.
• Blolabel, OCR read thirteen iPhone screens and the accessibility tree already had everything on eleven of them, 16 September 2026.