Xiaomi-GUI-0 trains on physical phones and still scores 72.0% on its own harder benchmark against 78.9% on AndroidWorld; phoneshell's board sits at 89.7% to 96.6% with nothing to separate it

Training closed-loop on real hardware was supposed to close the gap between a benchmark score and what a phone agent can actually do. It didn't close it inside Xiaomi's own numbers. The public benchmark still wins by 6.9 points, and phoneshell's own board shows the same ceiling from the other side.

Xiaomi-GUI-0, a 32-author technical report posted 30 June 2026 and revised 1 July (arXiv:2606.31410), trains a vision-language GUI agent through three stages, supervised fine-tuning, step-level reinforcement learning and agentic reinforcement learning, run with physical phones as the primary execution environment and sandboxes only as auxiliary support. The paper's own argument for doing this is that offline trajectories, simulated environments and standardized benchmarks differ from real apps in layout, interaction logic and abnormal-state distribution, which opens a gap between benchmark scores and real-world usability. Its reported numbers do not close that gap in the direction the argument implies: 72.0% on RealMobile, its own in-house benchmark built to resemble real user traffic, against 78.9% on AndroidWorld, the public benchmark the rest of the field is tuned to. The harder, more realistic benchmark, the one the physical-device training loop was built for, scores 6.9 points lower.

What changed, and when

Xiaomi's report describes a closed loop: data collection, training, rollout and evaluation all run with an execution distribution close to real deployment, meaning physical devices rather than emulators carry the weight at every stage. Training data covers high-frequency tasks, long-tail intents, and an error-driven data flywheel that turns failed trajectories into corrected actions and recovery demonstrations. That is a materially different approach from training mainly on offline logs or in a simulator and evaluating once at the end, and the paper frames it as the fix for exactly the gap this article is about. The two numbers it reports, 72.0% on RealMobile and 78.9% on AndroidWorld, are the only quantitative comparison the abstract gives, and the abstract does not state a reason for the gap beyond its general argument about standardized benchmarks differing from real apps.

phoneshell's own board runs the opposite experiment on the opposite platform: six frontier models from six vendors, Kimi K3, GPT-5.1, Claude Opus 4.8, Qwen3-Max, Gemini 3.1 Pro and DeepSeek v3.2, driven over the same 58 iOS tasks on a physical iPhone 14 Pro, 349 scored cells, $9.25 of API spend. Every task is a first-party Apple app with a clean accessibility tree. The result, covered here on 10 September as the board that cannot rank, is a tight 89.7% to 96.6% band in which not one of the fifteen possible pairs between models separates under McNemar's exact test. Forty-six of the 58 tasks were passed by every model. The suite is at ceiling for the same reason phoneshell predicted it would be: well-labelled, standardized, first-party interfaces are the easy case, for any model, trained any way.

Two benchmarks, two ceilings, opposite directions

What it means for someone reading a phone-agent benchmark score

The honest reading of Xiaomi's two numbers is not that physical-device training failed. It is that training method and benchmark difficulty are separate variables, and a paper that moves the first one does not thereby move the second one in a predictable direction. AndroidWorld is a known quantity that much of the field has spent two years optimizing against; RealMobile is Xiaomi's own construction, built specifically to carry the diversity and mess the abstract says public benchmarks lack. A model can get measurably better at the hard thing, through exactly the training loop the paper describes, and still score lower on it than on the easy thing, because the easy thing started easier and stayed easier. Nothing in the abstract licenses a claim that RealMobile's 72.0% would have been lower without the physical-device loop; the report has no such counterfactual to compare it against.

phoneshell's board makes the matching point from the other side, with a statistical test behind it rather than a single pair of numbers. First-party iOS apps, built by Apple, with accessibility trees Apple controls, are the standardized surface. Six independently trained frontier models land in a 6.9-point band on that surface and no pair of them separates. That is not six models converging on a hard problem; it is six models all finding the same easy problem easy, the mirror image of a benchmark with nothing left to discriminate. Put the two findings together and the shape is consistent: whichever side of the real-device-versus-standardized-surface line a task sits on predicts its difficulty for a competent model far better than which lab trained the model or how.

• A single benchmark score, in isolation, says almost nothing about how a model will do on a different distribution of apps. The number that matters is the gap between a model's easy-surface score and its hard-surface score, and Xiaomi's own report is one of the few places that gap is published at all for a physical-device-trained agent.
• phoneshell's board cannot yet produce the matching comparison for iOS, because it has no equivalent to RealMobile, a deliberately harder, messier benchmark built from real app diversity rather than first-party apps. Building one is the gap this article actually points at.
• "Trained on physical devices" is doing less work in the abstract than it appears to. It describes where training happened, not how hard the evaluation surface was. Both numbers can be true and the training method can still be a real improvement over a simulator-only baseline that Xiaomi's report simply does not measure.

Check it yourself

# Xiaomi-GUI-0's abstract and reported numbers
curl -s https://arxiv.org/abs/2606.31410

git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n -A20 "## 30. Adding tasks moved the ranking further out of reach" FINDINGS.md
python3 -c "import json; d=json.load(open('site/results.json')); print(d['board'])"

What would prove this wrong

• We read the 6.9-point gap as evidence that AndroidWorld remains the more standardized, more saturated surface even against a model trained specifically on physical-device diversity. That is wrong if Xiaomi, or anyone reproducing RealMobile from a published task list, shows a general-purpose model with no physical-device training scoring at or above 72.0% on it; that would mean RealMobile is not harder for the reason this article assumes, just differently distributed. Untested as of this writing, falsifiable by whoever runs it, through 31 December 2027.
• We predict that if phoneshell builds an equivalent harder benchmark for iOS, drawing on third-party apps with the kind of opaque accessibility trees already documented in FINDINGS.md section 23 and section 26, its six-model board's scores would drop well below the current 89.7% to 96.6% band and the fifteen pairs would very likely start separating, because disagreement, not raw difficulty, is what a ranking needs. We will update this article once that benchmark exists and the sweep has run.