52 of MobileGUI-VBench's 150 tasks require the agent to stop and ask, and models that never ask top out near 65

An original count over vivo's released benchmark file, the report's own score table, and the judge ablation that moved RL success from 72.52 to 77.42 with the reward unchanged.

52 of the 150 tasks in vivo's MobileGUI-VBench, released with the BlueLM-GUI report on 11 September 2026, are graded on whether the agent stops to ask or hands control back before it sends, orders or deletes. An agent that never asks can pass at most the other 98, a ceiling of about 65.3. Nine of the 13 outside models vivo reports score 66.4 or less, including the three best at plain execution on AndroidWorld. BlueLM-GUI, trained to call the user, scores 87.4 while trailing Qwen3.8-Max on AndroidWorld, 84.9 to 85.3.

What vivo released, and what it did not

vivo AI Lab posted BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents as arXiv:2609.12394 on 11 September 2026 (revised 14 and 15 September, 43 authors). The model is a 35B-A3B mixture of experts built on Qwen3.5-35B-A3B through continual pre-training (15B tokens, 64 GPUs), supervised fine-tuning (5B tokens, 32 GPUs) and GRPO reinforcement learning (32 GPUs, 8 rollouts for each of 8 instructions per iteration, at most 20 steps). The GitHub repository vivo-ai-lab/BlueLM-GUI carries the report, the benchmark and an AndroidWorld harness. It does not carry weights: the README lists the model's access as "In-house". The benchmark is public, as a 150-line JSONL on GitHub and as vivo-ai/MobileGUI-VBench on Hugging Face (last modified 15 September).

We counted the clarification_requirement field in that JSONL on 21 September. 98 tasks leave it empty. 52 set it: 16 "clarify before sending", 6 "hand over before placing the order", 2 "clarify before deleting", 2 "request handover at the login page", 2 "stopping at the login page is enough", and 24 more specific variants, several of which combine a clarification with a handover before an order. 65 tasks carry a precondition the device farm must set up first. The queries are in Chinese, across 40 apps such as QQ, NetEase Cloud Music, Xiaohongshu, Bilibili, DiDi and Amap.

The shape is two clusters, not one ranking. Models from the Seed family, Gemini3.7-Flash and BlueLM-GUI sit above 77 on VBench. Every other model sits at or below 66.4, and the ones that are strongest at plain execution on AndroidWorld (Qwen3.8-Max, Qwen3.8-Flash, PhoneBuddy) fall furthest. The report reads the same table the same way: "Strong execution therefore does not carry over to the interaction and refusal behavior that MobileGUI-VBench asks for." Our count adds the size of that effect. Some requirements are lenient, "stopping at the login page is enough" and "may clarify before liking" among them, so 65.3 is an approximate ceiling rather than an exact one, and GLM5.3-Flash at 66.4 is just over it.

How the grading works, and the 4.9 points it moved

The benchmark gives no partial credit. On an interactive query, an agent that passes a decision point involving money or private data without calling call_user fails "even if the task appears to have been completed". Several fine-tuned VLM judges with different rule sets rule independently, a trajectory passes only if all of them agree, and split verdicts go to human experts. Failures are attributed first, and a trajectory broken by the environment (an expired login, a redesigned screen, a network error) is not scored: the device is reset and the task run again.

Training data goes through a second grader, the Heterogeneous Triple-System Consensus, and the RL reward through a third, a judge queried several times with a majority vote. Section 5.3.5 is the part worth reading twice. Replacing one holistic judgement with "disjunctive bypass rules, under which hitting any one condition was enough to pass" by a set of atomic conditions, each bound to a designated frame and all required, judged five times, raised RL success on MobileGUI-VBench from 72.52 to 77.42. The paper's own reading: "those 4.9 points do not come from a better objective. They come from measuring the same objective more accurately."

What it means for anyone comparing phone agents

A MobileGUI-VBench score is mostly two numbers added together: how often the agent finishes, and how often it stops at the right moment. A model tuned to finish tasks, which is what AndroidWorld rewards, is capped near 65 by construction. Report the two halves separately, or the benchmark will rank willingness to ask as if it were competence. That is not a criticism of the design: an agent that orders food without confirming is worse, and the benchmark is right to fail it. It is a reason not to read 87.4 against 64.2 as "better at phones".

The judge result generalises further than the model. A disjunctive rule, pass if any one condition holds, is the same defect we found in our own suite on 20 September: seven of phoneshell's 80 tasks assert only which app is in front, and three of those are satisfied by the first leg of a three-app instruction. vivo measured what a defect of that shape costs when it feeds a reward: 4.9 points of RL success. With 150 tasks, one task is about 0.67 points, so that is roughly seven tasks. It is also close to the 5.1 point margin the report claims over the best closed model, which is a reason to want that margin reproduced with a second judge before treating it as settled.

Two things in the report do not match each other, and a reader reproducing it should know both. The abstract says every rollout runs "on hundreds of real phones rather than sandboxes or emulators"; section 5.1 describes the RL environment as "hundreds of real devices and Android emulators", with no split given. And none of 72.52, 77.42 or 87.4 is a whole number of tasks out of 150, so those figures are averages over repeated runs or over a different query set. The report does not say which.

One design choice is worth flagging for its own sake. To keep commercial apps from blocking the farm, the devices are "bound to dedicated physical SIM cards on independent cellular networks", and touches are injected "with realistic physical acceleration, evading commercial apps' device risk-control engines". That is a training rig engineered to look like people to fraud detection. We note it without a verdict; it is the part of the method that a lab outside China, or outside a phone maker, is least likely to be able to copy.

Check it yourself

# the benchmark and the clarification count
curl -sL https://raw.githubusercontent.com/vivo-ai-lab/BlueLM-GUI/main/MobileGUI-VBench/MobileGUI-VBench.jsonl -o vbench.jsonl
python3 -c "import json,collections; r=[json.loads(l) for l in open('vbench.jsonl')];
print(len(r), sum(1 for x in r if x['clarification_requirement']));
print(collections.Counter(x['clarification_requirement'] for x in r).most_common(6))"

# the scores, the judge ablation and the emulator sentence
curl -sL https://arxiv.org/html/2609.12394v3 -o bluelm.html
grep -o '72.52\|77.42\|real devices and Android emulators\|rather than sandboxes or emulators' bluelm.html | sort | uniq -c

# weights: the README lists access as In-house
gh api repos/vivo-ai-lab/BlueLM-GUI/contents --jq '.[].name'

What would prove this wrong

The claim is that MobileGUI-VBench's spread between models comes mainly from the 52 tasks that require asking, not from execution skill. It is testable from vivo's own trajectories: split each model's score into the 98 plain tasks and the 52 ask-or-hand-over tasks. If Qwen3.8-Max and BlueLM-GUI are within 5 points of each other on the 98 plain tasks, the claim holds. If BlueLM-GUI leads there by more than 10 points, its lead is execution and our reading is wrong. We will request the per-task results through the repository's issue tracker and publish the split, or the lack of an answer, by 30 November 2026.