AndroidControl-Curated finds 24 to 38 percent of labels wrong per category; phoneshell's checker has no recorded action to be wrong about

Leung, Xi and Zuo re-audited AndroidControl's ground-truth action labels and sorted the damage into three categories covering 24.13%, 8.12% and 37.63% of reviewed samples. Fixing them lifts curated success rates 11 to 18 points for five re-tested models, all of which were being marked wrong for picking an action the original label never considered valid. phoneshell's 80 tasks cannot carry this specific failure, not because its labels are better audited, but because its checker never records a "correct" action to compare against in the first place.

AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification (arXiv:2510.18488, Leung, Xi, Zuo, submitted 21 October 2025) starts from a specific complaint about AndroidControl, the widely used Android action-grounding benchmark: its ground-truth labels were written once, by one annotator per episode, and never checked against the actual range of actions that would complete the task. The authors ran automated corrections, then had an expert panel review a sample of them, then had an independent auditor check the panel's calls. Three error categories came out of that process, reported as a share of reviewed samples: Unclear Task, 24.13%, where the instruction itself does not pin down one action; Multiple Valid Actions, 8.12%, where the recorded action was correct but not the only correct one, so a model choosing a different valid action was marked wrong; and Wrong Ground Truth, 37.63%, where the recorded label was simply incorrect. The paper's own text is inconsistent about the total: its introduction cites "approximately 30%" of the benchmark as flawed, while section 3.4.1 calls the combined category share "over 70%," and the three numbers above sum to 69.88%, matching neither figure exactly. We report both statements rather than resolve the discrepancy, since the paper does not explain it.

What purification is worth, model by model

Table 2 of the paper is the only place it compares original AndroidControl scores against the curated version for the same models, and it covers five: GUI-R1-3B, GUI-R1-7B, Infi-GUI-R1, Qwen3-VL-235B and Magma-R1, all on the Hard subset. Every one of them gained from fixing the labels, and the gains are not small: Qwen3-VL-235B rose from 61.2% to 76.5%, a gain of 15.3 points, and Magma-R1 rose from 57.6% to 75.3%, a gain of 17.7 points. The paper does not report an original-benchmark score for GPT-4o, Qwen3-VL-30B, OS-Atlas-4B, UI-R1 or any Gemini model, so those five appear only with their curated-only numbers in Table 1, not as part of this comparison.

Source: arXiv:2510.18488, Table 2, Hard subset, read 30 October 2026.

Why phoneshell's checker cannot have this error, by construction

AndroidControl's flaw is specific to one benchmark design: a human annotator performs a task once, records the action taken at each step as the label, and every model is then graded on whether it reproduces that recorded action. If the annotator's path was ambiguous, non-unique, or simply wrong, every model that disagrees with it is marked wrong too, regardless of what actually happened on the device afterward. bench/schema.py states phoneshell's own benchmark design in its module docstring: a task is three parts, "setup" (deterministic steps, no model involved), "instruct" (the sentence handed to the agent), and "checks" (assertions run afterward that decide pass or fail automatically), and the file is explicit that "the checker is the product." A Check dataclass carries a kind, resolved in bench/verify.py against live device state: a foreground app, an element present in the accessibility tree, a toggle's position, or, for the one task type with no device state to inspect, a scan of the agent's own stated answer. None of this is a comparison against a recorded action sequence. There is no field anywhere in Check for "the action the annotator took," so a model that reaches the same end state by a different path passes exactly the same as one that reproduces the original author's keystrokes.

What it means for choosing or building a mobile agent benchmark

• A benchmark that scores action-matching against one recorded trajectory needs exactly the kind of purification pass AndroidControl-Curated ran, and the 11 to 18 point swings in Table 2 say the uncorrected scores were not close approximations, they were measuring annotator idiosyncrasy as much as model skill.
• A benchmark that scores device state after the fact, phoneshell's design, sidesteps this category of error but does not replace the need for review: its own setup scripts and check assertions can still be wrong, just not wrong in this specific way. The two designs trade one failure mode for a different one, they are not simply better or worse along the same axis.
• The paper's own unresolved gap, "approximately 30%" versus "over 70%" for what sounds like the same quantity, is itself a small instance of exactly the problem it is diagnosing: a reported number that nobody re-audited before publication. Treat the Table 2 figures, which are per-model and reproducible from the released data, as the load-bearing claim here, not the prose summaries.

Check it yourself

# phoneshell's own statement of what a task and a check are
sed -n '1,20p' phoneshell/bench/schema.py

# the Check dataclass: confirm there is no recorded-action field
grep -n "class Check" -A 20 phoneshell/bench/schema.py

# how a Check is resolved against live device state, not a recorded path
sed -n '1,20p' phoneshell/bench/verify.py

# AndroidControl-Curated's own error-category table
curl -s https://arxiv.org/html/2510.18488 | grep -iE "Unclear Task|Multiple Valid|Wrong Ground Truth"

# AndroidControl-Curated's Table 2, original vs curated
curl -s https://arxiv.org/html/2510.18488 | grep -iE "Table 2|GUI-R1|Magma-R1"

What would prove this wrong

We claim phoneshell's Check dataclass has no field recording a reference action sequence, only a kind resolved against live device state, which is checkable directly against bench/schema.py and would be false the moment such a field is added. We also claim the five models in AndroidControl-Curated's Table 2 gained 11.2 to 17.7 points purely from label correction on an unchanged action space, which rests entirely on the paper's own methodology being sound; a later audit finding the correction process itself introduced new errors, something the paper does not rule out given its own unreconciled 30%-versus-70% figures, would undercut the size of the claimed gain without changing the direction of it. We will revisit this piece if either phoneshell's checker design changes to include trajectory matching, or a follow-up paper re-audits AndroidControl-Curated's own corrections, by 30 April 2027.