AndroidDaily's GRADE judges closed-source apps with 87.37% human agreement, which is also a 12.63-point disagreement rate on 350 tasks; phoneshell's checks disagree with nobody because 133 of 153 assertions never ask a judge

AndroidDaily sets 350 tasks on 94 real, closed-source Android apps that expose no internal state to check against, and solves the verification problem with GRADE, a tiered-guideline judge reporting 87.37% agreement with human evaluators. The same arithmetic says GRADE disagrees with a human on roughly one task in eight. phoneshell faces the identical closed-source-app problem on real consumer iOS apps and answers it differently: of its 153 assertions over 80 tasks, 133 read a bundle ID, an accessibility-tree match or a regex pattern straight from the device, and only 20 ever ask a model to judge anything at all.

What changed, and when

AndroidDaily was posted to arXiv as 2605.27761 on 26 May 2026 by a 17-author team led by Yifan Sui, setting 350 "realistic daily-use tasks" across 94 high-frequency Android apps spanning transportation, shopping, local services, entertainment, content creation, social media and everyday utilities. The paper's stated motivation is that these are real, closed-source apps: they don't expose internal state, so the database-read or API-call verification that simulator-based benchmarks rely on simply does not exist here. Its answer is GRADE, Guideline-grounded Reviewer for Automatic Diagnostic Evaluation, a process-aware judge that walks an agent's visual trajectory against three tiers of observable external guidelines, operational obligations, output quality and negative constraints, and produces step-level diagnostic judgments rather than one pass or fail stamp at the end.

GRADE reports 87.37% agreement with human evaluators, and the strongest model tested on AndroidDaily reaches a 62.0% success rate. Stated the other way, 87.37% agreement is also a 12.63% disagreement rate, and applied to AndroidDaily's own 350-task set that works out to roughly 44 tasks (350 x 0.1263 ≈ 44.2) where GRADE's own verdict would not match a human reviewer looking at the same trajectory. The paper reports the 87.37% figure as a validation of the judge, which it is; it is also, arithmetically, a built-in error bar on every success rate the judge subsequently reports, including the 62.0% headline number.

Where phoneshell narrows judgment to 20 of 153 assertions

phoneshell/bench/verify.py implements twelve check kinds, but the 80 published tasks only ever assert five of them, for a total of 153 assertions: foreground_app (76 uses, reading WDA's activeAppInfo and comparing a bundle ID string), element_exists (31, a text match against the current accessibility tree), regex_on_screen (23, a regex run over every element's joined text), answer_contains (20), and element_exists_anywhere (3, the same tree match plus up to six scroll passes). Seven more kinds, text_on_screen, element_value, switch_on, switch_off, screen_matches, ocr_contains and app_state, are implemented in the same file but asserted by zero tasks, according to the project's own task-validation script.

Of those five asserted kinds, four read the phone directly and cannot disagree with a human the way GRADE can, because there is no judgment step between the device and the verdict: a bundle ID either matches the string a task wants or it does not. The fifth, answer_contains, is explicit in the module's own docstring about why it exists at all: "the ONLY check that grades what the model said rather than what the phone shows," reserved for the narrow case where a task's deliverable is information with no phone state to inspect, such as reading a word out of a picture. 133 of 153 assertions, 86.9%, never ask anything to be judged. The other 13.1% ask a single yes-or-no substring question of the model's own sentence, not a tiered guideline rubric scored against a visual trajectory.

What it means for anyone grading agents on real apps

• Scaling judgment up buys coverage a pure state check cannot reach, such as whether a written summary is actually good, not just present, which is exactly what GRADE's "output quality" tier is for. That coverage comes with a built-in, measured error rate: 12.63% here, reported honestly by the paper rather than buried, but still a rate that applies to every downstream number GRADE produces.
• Narrowing judgment to zero wherever device state is available, phoneshell's approach, buys certainty on the 86.9% of assertions that use it, at the cost of being unable to grade anything that is not representable as a bundle ID, an on-screen string, or a regex pattern. A task like "write a good caption" has no foreground_app check that could ever grade it; phoneshell's 80-task set does not currently contain a task that would need one.
• The two approaches are not actually in competition for the same 20 phoneshell assertions that already use answer_contains: those are the tasks with no device state to read, the same category GRADE's existence implies should exist on AndroidDaily's closed-source apps too, just at a scale (350 tasks across many closed-source apps) where a bespoke page-hosted ground truth per task, phoneshell's own workaround, would not hold up.
• Seven of phoneshell's own twelve implemented check kinds, including switch_on, switch_off and app_state, sit unused by any of its 80 tasks. The codebase does not document why; it is a real gap in this check's own evidence, not a claim either way about whether those checks are needed.

Check it yourself

# AndroidDaily's own agreement figure and what it implies about disagreement
curl -s https://arxiv.org/abs/2605.27761 | grep -o "87\.37%\|62\.0%"
python3 -c "print(round(350 * (1 - 0.8737)))"   # roughly how many of 350 tasks GRADE would score differently from a human

# phoneshell's own check-kind usage, straight from the task files
git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "if kind ==" phoneshell/bench/verify.py             # twelve implemented kinds
python3 scripts/validate_tasks.py --count-check-kinds 2>/dev/null || grep -rho "kind: [a-z_]*" tasks/*.yaml | sort | uniq -c

What would prove this wrong

We claim AndroidDaily's reported 87.37% agreement figure is arithmetically equivalent to a 12.63% disagreement rate, which on its own 350-task set is roughly 44 tasks; that arithmetic is wrong only if the 87.37% figure was computed on a different denominator than the full 350-task set, which the abstract does not specify and which we could not verify further. We claim phoneshell's verify.py asserts 133 of 153 checks without any judgment step; that is wrong the moment a sixth check kind that involves a model judgment call gets asserted by a phoneshell task, and none did as of 25 October 2026.