PhoneHarness's abstract says its mixed GUI/CLI/Tool agent reaches 75.0%; its own model-combination table gives the same configuration 74.8%; phoneshell has no Tool action type to even build that table with

PhoneHarness mixes GUI, CLI and Tool actions into one Android agent and reports a 75.0% headline pass rate, 12.9 points ahead of the strongest non-PhoneHarness setting. Its own model-combination table lists 74.8% for the DeepSeek V4 flash plus Seed2.0-Pro pairing the text says produced that headline number, a small but real mismatch inside one paper. phoneshell's verify.py has no check that could even grade two of PhoneHarness's six side-effect categories, tool-called and email-recipient, because phoneshell has no Tool action type for those checks to apply to.

What changed, and when

PhoneHarness was posted to arXiv as 2606.14832 on 12 June 2026 by a 21-author team led by Chenxin Li, introducing a mixed-action execution harness for Android phone-use agents built around three action types: GUI (taps, swipes, text input, UI-tree retrieval, delegated to a GUI model), CLI (deterministic ADB/shell/Python commands for device-state queries, wake-lock control, file downloads) and Tool (host-side MCP-style utilities: search, email, document and file processing, chart generation, calendar creation). Its benchmark, PhoneHarness Bench, has an annotated evaluation split of 124 tasks drawn from a 181-task candidate pool: 30 device/system tasks, 30 single-app GUI tasks, 35 tool-assisted tasks and 29 cross-app tasks.

The paper's headline claim is a 75.0% pass rate on that 124-task split, 12.9 points ahead of the strongest non-PhoneHarness setting (62.1%, tied between the MobileClaw harness-based controller and the Seed2.0-Pro model-only agent) and 37.9 points ahead of AutoGLM-Phone, a native phone-use GUI agent, at 37.1%. The same paper's model-combination table, listing results by which model plays which role in the harness, gives 74.8% for DeepSeek V4 flash paired with Seed2.0-Pro as the GUI worker, the configuration the surrounding text identifies as the one behind the 75.0% headline. A 0.2-point gap between a paper's stated headline and its own supporting table, for what the text describes as the same run, is the kind of thing a reader citing only the abstract would never notice.

Two check kinds phoneshell cannot build without a Tool action type

PhoneHarness verifies its mixed-action tasks with six rule-based, trace-backed checks: whether a required tool was called, whether an email went to the correct recipient, whether a device setting reached the expected value, whether a generated artifact exists and meets size or content constraints, whether a calendar event was created, and whether the final answer contains required information. phoneshell's verify.py, read in full, implements twelve check kinds but asserts only five across its 80 tasks: foreground_app (76), element_exists (31), regex_on_screen (23), answer_contains (20) and element_exists_anywhere (3). None of the five, and none of the seven unused kinds either, checks whether a tool was called or whether an email reached a recipient, because phoneshell's action module (phoneshell/actions.py) has no Tool action type and no email-sending capability for such a check to apply to in the first place. The verification surface in both projects tracks the action surface: PhoneHarness needed six check kinds because it gave its agent three different ways to act; phoneshell's five asserted kinds all grade GUI-only state because GUI is the only thing phoneshell's agent does.

Where the two projects' taxonomies do overlap, the comparison is closer than it looks. PhoneHarness's "device setting reached the expected value" check has a direct phoneshell analogue in switch_on, switch_off and app_state, three kinds implemented in verify.py but asserted by zero of phoneshell's 80 published tasks. PhoneHarness's "final answer contains required information" check is, in method if not in wording, the same operation as phoneshell's answer_contains: both grade a model's own sentence against a required substring because the task's deliverable is information rather than a state change. Neither project's final-answer check reads the device at all; both projects reserve that one exception for the same reason.

What it means for anyone comparing verification taxonomies

• A benchmark's check-kind count is mostly a function of its action-type count, not an independent design choice. PhoneHarness needed a tool-called check and an email-recipient check because it gave its agent tools and email; phoneshell has neither kind because it gave its agent neither capability. Comparing the raw count of check kinds between two projects without naming which actions each kind grades will overstate how different the two verification philosophies actually are.
• The 75.0%-versus-74.8% gap is small enough to be a rounding or re-run artifact, and the paper offers no note explaining it either way. That is the same category of problem as the iOSWorld arXiv-versus-ICML gap covered separately on this site: small, real, unexplained, and invisible to a reader who checks only one of a paper's own numbers for a given result.
• phoneshell's seven unused check kinds, switch_on, switch_off, element_value, screen_matches, ocr_contains, app_state and text_on_screen, are a measurable amount of unused verification capacity: 58% of the implemented kinds (7 of 12) have never been exercised by a real task in 80 attempts, a gap any future phoneshell task author should know about before assuming an unused kind is untested as well as unused.
• PhoneHarness's 35-of-124 tool-assisted tasks and phoneshell's 0-of-80 tool-involving tasks are not a performance gap, they describe two agents built to do different amounts of stuff. Neither number says anything about which agent is better at the actions it actually has.

Check it yourself

# PhoneHarness's own two numbers for the same configuration
curl -s https://arxiv.org/abs/2606.14832 | grep -o "75\.0%\|74\.8%"

# phoneshell has no Tool action type and no email/MCP-tool verb
git clone https://github.com/Blomega/phoneshell && cd phoneshell
grep -n "def \|action ==" phoneshell/actions.py | grep -iE "email|tool|mcp"   # nothing matches
grep -n "if kind ==" phoneshell/bench/verify.py                               # twelve kinds, read which ones exist

What would prove this wrong

We claim PhoneHarness's abstract (75.0%) and its model-combination table (74.8%) report two different numbers for what the surrounding text identifies as the same DeepSeek V4 flash plus Seed2.0-Pro configuration. That is wrong if a revision of arXiv:2606.14832 clarifies that the two figures describe different runs or different rounding bases; we found no such clarification as of 25 October 2026. We claim phoneshell's action module has no Tool action type and therefore no tool-called or email-recipient check. That is wrong the moment such an action and check appear in phoneshell/actions.py and phoneshell/bench/verify.py, and we found neither as of the same date.