MobileGym's simulation-trained policy keeps 95.1% of its gain on a physical Android phone, a 2.1-point loss out of 42.8; phoneshell trains and grades on physical hardware from the first run, with no simulated stage to lose anything from

MobileGym trains Qwen3-VL-4B-Instruct with reinforcement learning inside a parallel, deterministic Android simulator, then checks the same 59 tasks on a physical phone. The simulated gain is 42.8 points; the real-device gain is 40.7. That is a 95.1% retention rate, and a 4.9% loss. phoneshell's own 80 tasks never touch a simulator in the first place, which is a different trade, not a better number.

MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research (arXiv:2605.26114), submitted 25 May 2026 and revised most recently 23 September 2026 (v3) for EMNLP 2026, by Dingbang Wu, Rui Hao, Haiyang Wang, Shuzhe Wu, Han Xiao, Zhenghong Li, Bojiang Zhou, Zheng Ju, Zichen Liu, Lue Fan, and Zhaoxiang Zhang, builds a browser-hosted simulator across 28 Android apps that judges tasks deterministically from structured JSON state rather than from pixels or model self-report. Each simulated instance costs about 400 MB of memory and about 3 seconds to cold-start, so a single server can host hundreds of them in parallel, which is the entire economic case for simulating rather than running on physical hardware: training runs that would need hundreds of phones instead need hundreds of lightweight instances on one machine. The paper then asks the question that economic case depends on: does a policy trained inside that simulator still work once it is moved to an actual phone.

The training gain mostly survives the jump to hardware, and the loss is now a number

On MobileGym-Bench's full 256-task simulation test set, GRPO training raises Qwen3-VL-4B-Instruct's overall success rate from 9.4% to 22.2%, a gain of 12.8 points, entirely inside the simulator. The paper then isolates a 59-task subset chosen for being reproducible on real hardware (excluding 8 of an original 67 "signal-bucket" tasks that depended on unreproducible account state or an irreversible action), and runs that same comparison twice: once inside the simulator and once on a physical Redmi Note 12 Turbo. On the simulator side of that 59-task subset, training raises success from 33.9% to 76.7%, a gain of 42.8 points. On the real device, the same training raises success from 32.2% to 72.9%, a gain of 40.7 points. Dividing one gain by the other, 40.7/42.8, gives the paper's own reported retention figure: 95.1%.

Two different bets on the same 4.9% gap

MobileGym's architecture is built to make the simulator the default and the physical device the spot check: hundreds of 400 MB instances per server, 3-second cold starts, and a 59-task real-hardware pass used only to validate that training, which happened in the simulator, actually transfers. phoneshell's architecture answers a different question and makes the opposite bet. Its houseFacts describe 80 public tasks run on one physical iPhone 17 Pro Max on iOS 26.6, with no simulated stage at all; every number phoneshell publishes, including its own screenshot-cost and pixel-only-task figures, comes from the device, never from a simulated stand-in for it. MobileGym's 95.1% retention is evidence that, for its own task mix, the bet on simulating first and validating later mostly pays off. It is also evidence that mostly is not entirely: 4.9% of the simulated gain, about 2.1 of the 42.8 points, did not survive contact with a real Android build, real app data, and the real-world entities (contacts, points of interest) the paper names as sources of divergence. Whether that gap matters depends on whether the failures concentrated in it look like noise or like exactly the cases a benchmark exists to catch, a question this paper's 59-task sample is not large enough to answer on its own.

What it means for choosing simulation, hardware, or both

• 95.1% retention is a specific, falsifiable number for a specific policy, model size, and 59-task sample. It licenses the claim "this simulator's training signal mostly transfers for Qwen3-VL-4B-Instruct on these tasks," not the broader claim that simulation is interchangeable with hardware for mobile agents in general.
• The 4.9% that did not transfer is the part a parallel simulator cannot, by MobileGym's own account, currently see: divergence from real app data, real-world entities, and UI details the simulator does not model. A benchmark built entirely on physical hardware, like phoneshell's, cannot lose that signal to a sim-to-real gap, because there is no simulated stage to diverge from.
• The two designs are optimizing different things. MobileGym optimizes for training throughput, hundreds of parallel instances at 400 MB each, and spends a 59-task physical check to bound its own error. phoneshell optimizes for the error bound being zero by construction, at the cost of never getting that parallel throughput at all.

Check it yourself

# MobileGym's paper and sim-to-real section
curl -s https://arxiv.org/abs/2605.26114
# full results (section 5.2, Table 2, the 59-task real-device subset): https://arxiv.org/html/2605.26114

# phoneshell's device-only design: no simulator anywhere in the grading path
grep -rln "simulator\|simulated" phoneshell/ 2>/dev/null
grep -n "physical\|iPhone 17" phoneshell/bench/runner.py phoneshell/config.py 2>/dev/null

What would prove this wrong

We report MobileGym's real-device retention as 95.1%, computed from a 42.8-point simulated gain and a 40.7-point physical gain on the same 59-task subset, both read directly from arXiv:2605.26114 section 5.2. That is wrong if a future revision of the paper restates either figure, or if the 59-task subset is later shown to be unrepresentative of the full 256-task benchmark in a way the authors have not flagged. We also claim phoneshell's physical-only design cannot have a sim-to-real gap because it has no simulated stage; that framing would be wrong if phoneshell's harness is later shown to rely on any simulated or mocked component for grading rather than live device state. We will check both by 31 March 2027.

Sources

• Wu, Hao, Wang, Wu, Xiao, Li, Zhou, Ju, Liu, Fan, Zhang, MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research, arXiv:2605.26114, submitted 25 May 2026, revised 23 September 2026 (v3), EMNLP 2026, read 18 October 2026.
• Full results: arxiv.org/html/2605.26114 (section 5.2, Table 2, 59-task real-device subset).
• Blolabel, phoneshell: houseFacts and phoneshell/config.py, read 18 October 2026.