BlueLM-GUI trains on hundreds of real phones and 5,000 labelled failure samples; phoneshell runs one

Both projects agree that a simulator is not enough. BlueLM-GUI's answer is a fleet feeding a model. phoneshell's answer is a single iPhone feeding a leaderboard. Neither is a smaller version of the other; they keep different things from the same conviction.

The BlueLM-GUI technical report, submitted to arXiv on 11 September 2026 by Tong Ye and 42 co-authors, describes training a 35B-A3B mobile GUI agent on rollouts from "hundreds of real phones," with roughly 5,000 targeted samples covering 14 classes of anomalous device state folded back into training by what the paper calls an Error Correction and Derivation Module. phoneshell drives exactly one physical device, an iPhone 17 Pro Max on iOS 26.6, and keeps one row per attempt in site/results.json: a pass/fail verdict, a turn count, a cost, and the checks that ran. No phoneshell run has ever fed back into training anything, because phoneshell does not train a model at all. The two projects share a premise, real hardware over simulation, and solve nothing like the same problem with it.

Three principles, one loop

BlueLM-GUI's authors organize the system around three stated principles. "Every Sample Matters" is a dual-track data pipeline with multi-system evaluation, built around the Error Correction and Derivation Module: rather than discarding a trajectory where the agent failed, the module turns it into usable training signal, including the roughly 5,000 samples across 14 anomalous-state classes cited above. "Every Rollout Is Real" is the training recipe itself: continual pre-training, supervised fine-tuning, and agentic reinforcement learning, run, in the paper's words, on "hundreds of real phones," which the authors contrast with sandbox training on the grounds that a sandbox causes "a distribution mismatch with production environments." "Every Query Evolves" is the evaluation side: a benchmark the authors describe as "quota-driven" across three axes, scored by what they call "Heterogeneous Triple-System Consensus," designed to be upgraded as the model improves rather than saturate the way a fixed benchmark does. The result the report leads with is MobileGUI-VBench at 87.4, which it says surpasses the best closed-source model by 5.1 points, putting that closed-source figure at roughly 82.3, plus AndroidWorld at 84.9, described as the best score among open-source models.

Read end to end, the loop is: real device rollout, in bulk, across a fleet; failure classified into one of 14 known anomaly types rather than thrown away; that classified failure becomes training data for the next model version; the benchmark used to measure the next model version is itself allowed to get harder. Nothing about phoneshell's own real-device methodology does any of that, and nothing in phoneshell's stated purpose calls for it to.

What phoneshell keeps from a run

phoneshell's own real-device methodology runs one iPhone 17 Pro Max on iOS 26.6 through its 80-task suite and records exactly the fields visible in site/results.json: task_id, model, passed, skipped, skip_reason, turns, seconds, cost_usd, checks, scaffold, started_at. There is no field for a screenshot trail, no raw action log, and no classification of why a failed task failed beyond whatever a human reads off the checks array by hand. A failed phoneshell task is not an anomaly sample; it is a line in a leaderboard. Nothing downstream of that line retrains a model, because phoneshell does not ship a model, it grades whichever one a user points at the phone. That is a complete and defensible design for an evaluation harness, and it is also the reason the comparison with BlueLM-GUI is not "phoneshell is smaller," it is "phoneshell and BlueLM-GUI keep different things on purpose."

What it means for anyone reading a fleet number

• "Hundreds of real phones" and "one real iPhone" answer different questions. BlueLM-GUI's fleet exists to generate enough rollouts, including enough failures, to move model weights. phoneshell's single device exists to generate a trustworthy pass/fail number for whichever model is already trained. Neither fleet size tells you anything about the other project's rigor.
• The 14-class anomaly taxonomy is the part phoneshell most conspicuously lacks, and the part most worth borrowing regardless of scale: even on one device, grouping phoneshell's own failures by cause, rather than leaving each as an unlabelled passed: false row, would turn its results.json into something closer to a diagnosis than a scoreboard.
• A "quota-driven benchmark" that upgrades as the model improves is the opposite design choice from phoneshell's fixed 80-task suite. That is appropriate for BlueLM-GUI, where the benchmark exists to keep pace with a model under active development, and appropriate for phoneshell, where the benchmark exists to let the same 80 tasks compare different models on equal footing over time.
• Readers should not average the 87.4 and the phoneshell leaderboard's own pass rates into a single "how good is mobile AI" number. They are not measuring the same thing, were not run on the same tasks, and in BlueLM-GUI's case were not run against models phoneshell's leaderboard currently includes.

Check it yourself

# what a phoneshell run actually records
python3 -c "import json; d=json.load(open('site/results.json')); print(list(d['runs'][2].keys()))"
grep -n "houseFacts" -A10 content-engine/ios-agents.json

# confirm phoneshell has nothing resembling a training or retraining step
grep -rliE "fine.?tun|reinforcement learning|training data|checkpoint.pt" phoneshell/ || echo "no matches"

# BlueLM-GUI's own numbers
curl -s https://arxiv.org/abs/2609.12394 | grep -i "hundreds of real phones\|MobileGUI-VBench\|AndroidWorld"
# full report: https://arxiv.org/html/2609.12394

What would prove this wrong

We claim phoneshell records no information from a run that could feed back into retraining any model, and that it runs on exactly one physical device. The first is wrong if any future commit to phoneshell/bench/runner.py or its results schema adds a field for a classified failure cause or a stored trajectory; the second is wrong if the project's houseFacts are updated to describe more than one physical rig. We also derived BlueLM-GUI's "best closed-source" figure of approximately 82.3 from its stated 5.1-point gap rather than reading it directly off a table; that derivation is wrong if the full paper states a different closed-source baseline score. We will check content-engine/ios-agents.json's houseFacts and the BlueLM-GUI repository, if one is published, by 31 January 2027 and revise this comparison if either has changed.

Sources

• Ye et al. (43 authors), BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents, arXiv:2609.12394, submitted 11 September 2026, revised 15 September 2026 (v3), read 15 October 2026.
• Full text: arxiv.org/html/2609.12394.
• Blolabel, phoneshell: site/results.json, content-engine/ios-agents.json, read 15 October 2026.