Claude Opus 4.6 ranks first on task success and fourth on privacy across MyPhoneBench's 300 tasks; phoneshell's checks have no second axis to rank anything on

MyPhoneBench, the benchmark introduced in "Do Phone-Use Agents Respect Your Privacy?" (arXiv:2604.00986), scores five frontier models on 300 tasks across 10 mock apps twice: once for whether the task got done, once for whether the agent over-disclosed data nobody asked for. The two rankings do not match. phoneshell's own check vocabulary, nine kinds in total, has nothing that scores the second thing on any task whose author did not think to ask for it.

MyPhoneBench, the benchmark introduced in "Do Phone-Use Agents Respect Your Privacy?" (arXiv:2604.00986, submitted 1 April 2026 by a 22-author team including Zhengyang Tang, Ke Ji and Benyou Wang), ran five frontier models against 300 tasks across 10 mock consumer apps and scored each model on two separate axes: whether the task was completed, and whether the agent handled the user's personal data the way a careful human would. Claude Opus 4.6 finishes first on the first axis, completing 82.8% of tasks. On the second axis it finishes fourth, averaging 68.4% on the paper's privacy score. Once the two are combined into a single "privacy-qualified" success rate, the ranking changes: Qwen 3.5 Plus moves from second to first, and Claude Opus 4.6 drops to second. phoneshell's own check vocabulary, the Check kinds enumerated in bench/verify.py, has no concept that could produce a privacy score for anything. It has exactly one axis: whether the task's own author wrote a check for a given behaviour, and nothing checks what nobody thought to ask about.

What MyPhoneBench measured, and what moved the ranking

The benchmark's 10 mock apps (mZocdoc, mCVS, mDoorDash, mBooking, mZillow, mGEICO, mOpenTable, mDMV, mThumbtack, mEventbrite) are built specifically to make privacy failures observable: an optional insurance-ID field placed right after a required insurance-provider field, to test over-permissioning; a "Priority Scheduling" step that re-asks for a phone number already given, to test trap resistance; an optional date-of-birth field between two required fields, to test form minimization. The paper's own text calls form minimization the hardest probe for every model tested, and the single most persistent failure is agents filling in optional personal fields the task never required.

Source: "Do Phone-Use Agents Respect Your Privacy?", arXiv:2604.00986, read 27 October 2026. 5 models, 10 mock apps, 300 tasks.

phoneshell's one axis

bench/verify.py defines a Check as one of nine kinds: answer_contains, foreground_app, element_exists_anywhere, element_exists or text_on_screen, element_value, switch_on or switch_off, screen_matches, ocr_contains, regex_on_screen, and app_state. Every one of them reads the real phone's state and compares it against a value the task's YAML file specifies. There is no kind that runs on every task regardless of what its author wrote, and no kind designed to catch an agent filling in a field, or disclosing a value, the task never asked for. A phoneshell task that asks an agent to book a restaurant table checks whether the table got booked. If the agent also typed a date of birth into an optional field nobody required, as MyPhoneBench's own form-minimization probe is built to catch, nothing in verify.py's nine kinds would notice unless the task's author specifically wrote an element_value check, with negate: true, against that exact optional field, on that exact task. A search of the repository and the scripts directory for any mention of privacy, over-disclosure or data minimization returns nothing.

What it means for grading an agent's data handling

• MyPhoneBench's finding is structural, not a complaint about any one model: success and privacy are different things to measure, and measuring only the first tells you nothing reliable about the second. Claude Opus 4.6's own numbers are the clearest case, first on one axis and fourth on the other, 300 tasks in.
• phoneshell's check system was built to answer "did the task get done," a narrower and more tractable question, and it answers it well: every check kind reads device state rather than the model's own account of itself. That design choice has no opinion, one way or the other, about data handling, because nothing in it was built to have one.
• Adding a second axis to phoneshell would not require new infrastructure. It needs either a check that runs unconditionally alongside a task's existing checks (the architecture already runs a list of checks per task in run_all), or a MyPhoneBench-style probe retrofitted onto phoneshell's own forms, the direct way to find out whether the models phoneshell currently ranks on task success alone would reorder the way Claude and Qwen reorder in the source table.

Check it yourself

# phoneshell's entire check vocabulary, nine kinds, all state-based
grep -n "if kind ==" phoneshell/bench/verify.py

# confirm there is no privacy, over-disclosure or minimization concept anywhere
grep -rniE "privacy|over.?disclos|minimi[sz]ation" phoneshell/ scripts/ 2>/dev/null

# confirm checks are per-task opt-in, not a global second pass
grep -n "def run_all" -A3 phoneshell/bench/verify.py

# MyPhoneBench's own per-model table
curl -s https://arxiv.org/html/2604.00986 | grep -iE "Claude Opus|privacy"

What would prove this wrong

We claim phoneshell's check vocabulary has no mechanism that scores an agent's data-handling behaviour independently of what a task's own author chose to check. That is wrong if a check kind we missed, in verify.py or elsewhere, already does this. We also claim the ranking reshuffle MyPhoneBench reports between task success and privacy, confirmed on 10 mock apps and 300 tasks, has not been tested on any of phoneshell's own 80 iOS tasks or on the specific frontier models phoneshell grades. That is falsified the day a MyPhoneBench-style negated check is added to even a handful of phoneshell's existing form-filling tasks and run against the models phoneshell currently ranks, and the ranking either holds or does not. We will treat that run, if it happens, as the direct test, by 30 April 2027.