DIBench and PAIR grade which item a phone agent picked; 12 of phoneshell's 80 checks pass before the agent has picked anything

Two papers this week score the choice a phone agent makes, not whether it finished. We audited phoneshell's 80 device-graded tasks for the same question and found the suite cannot see a choice at all.

Two mobile-agent papers posted this week grade the item an agent picked rather than whether it finished. DIBench shows deceptive screen content raising an agent's completion rate from 42.0% to 72.4% while steering 45.8% of its choices to the attacker's target. PAIR shows subgoal success 7.0 to 15.4 points lower wherever the screen depends on whose phone it is. phoneshell's 80 device-graded tasks never ask an agent to commit to one item. Twelve of its checks pass on the screen the app opens to, or on the query the agent typed. In the 9 September run those twelve tasks passed 71 of 72 times, 71 of the 322 passes.

What the two papers measured

DIBench (Li Hu, Kanghua Mo, Yingbin Jin, Qingqing Ye and Haibo Hu, Hong Kong Polytechnic University; arXiv:2610.06898, dated 29 September 2026, listed 7 October) builds 1,000 clean and 36,672 injected selection tasks across 7 commercial and 3 simulated apps. Each task has an instruction, a candidate set and an acceptable set (the cheapest item, the highest-rated, an exact match). It scores three things separately: task completion rate (did the agent commit to any candidate), correct selection rate (was the choice acceptable), and attack success rate (did it pick the attacker's target while its paired clean run did not). Under the combined attack, completion went up for all four frameworks tested with Qwen2.5-VL on commercial screens, which is the paper's point: a completion score reads an attack as an improvement.

PAIR (Yeji Park, Jaeyun Shim and Taesik Gong, UNIST; arXiv:2610.07972, 6 October 2026) renders the same ten apps differently for ten personas drawn from PersonaLens, 120 tasks per persona, 1,200 in the test set. It labels each subgoal as user-agnostic (shared navigation) or user-conditioned (content that varies by user), and compares six agents. Task success across personas ranged from 43.8% to 53.1%. In user-conditioned contexts the typical failure was "selecting another item instead of the intended target", most often before the target had scrolled into view. RePAIR, its training method, raised user-conditioned subgoal success by 5.87 points and task success by 9.42 over its fine-tuned parent on unseen users.

Auditing phoneshell's 80 checks for a choice

phoneshell grades from the device: each task in environments/*.yaml lists checks that read the phone after the agent stops. The text checks use find_by_text in perception/tree.py, a lower-cased substring match over an element's label, name, value, identifier and child text. Value is where a search field keeps the typed query. We read all 80 tasks and asked of each one what is the earliest screen that satisfies every check. For Settings we could answer from the crawler's record of the root screen on the owner's iPhone (iOS 26.6.1, WebDriverAgent 16.12.4, 11 September), which lists the controls visible on load.

The four settings.search.* tasks say "open the page from the results". Their checks are satisfied twice over before that: by the root list, and by the query sitting in the search field. The five search tasks outside Settings ask for results to show, and their checks match the text the agent typed, so a search that returned nothing would pass. Not one of the 80 tasks hands the agent a candidate set and grades which candidate it took. In DIBench's vocabulary, phoneshell's checks sit below task completion: they cannot register a correct selection, a wrong one, or a steered one.

The storage banner is the PAIR effect turned around. PAIR finds agents failing on content that depends on the user. Here the grader passes on content that depends on the user: an account that has filled its iCloud plan puts the word "storage" on the root screen. The xv2 rows ran on an iPhone 14 Pro whose root screen we have not crawled, so for that phone the storage check passed through the typed query at least. The headline of that run, 322 of 349 or 92.3%, is 251 of 277 or 90.6% without the twelve. The order of the six models does not change.

What it means if you grade phone agents from the device

• Grade the destination, not a word. A Settings page has a navigation bar title. "The navigation bar says Storage" is a claim about where the agent ended. "Storage appears somewhere" is not.
• Exclude editable fields from text checks, or the agent's own typing grades itself. That costs one line in find_by_text for graders (skip SearchField and TextField values) and keeps it for the agent's own lookups.
• Run a null agent before trusting a task. One that only opens the target app, and one that only types the query, would have flagged all twelve tasks for the price of two short runs per task.
• Selection tasks need an acceptable set, as DIBench's do. On a phone that means recording the chosen item and its attributes (price, rating, title) at the moment of commitment. The phone's state after a tap rarely carries them.
• A personal phone is a persona of one. PAIR's 9.31-point spread across ten personas is the size of error a single-device suite cannot see, in either direction.

Check it yourself

# the matcher and the checks
sed -n '/def find_by_text/,/return hits/p' phoneshell/perception/tree.py
grep -A3 "^checks:" environments/settings_search_storage.yaml environments/books_search.yaml

# the run (task_id, passed)
curl -sLO https://huggingface.co/datasets/blolabel/phoneshell-bench/resolve/main/results/xv2.jsonl
python3 -c "import json;R=[json.loads(l) for l in open('xv2.jsonl')];S=('settings.search.wifi', 'settings.wifi.switch_visible', 'settings.search.battery', 'settings.search.bluetooth', 'settings.bluetooth.reach', 'settings.search.storage', 'settings.accessibility.reach', 'books.search', 'maps.search_place', 'notes.search', 'safari.search.web', 'contacts.search');a=[r for r in R if r['task_id'] in S];b=[r for r in R if r['task_id'] not in S];print(sum(r['passed'] for r in a),len(a),sum(r['passed'] for r in b),len(b))"
# expect: 71 72 251 277

# the papers
https://arxiv.org/abs/2610.06898   (Table 3, Commerce)
https://arxiv.org/abs/2610.07972   (Table 2)

The Settings root screen comes from the crawler's manifest on our rig (runtime/crawls/20260911-121256-settings/manifest.json, screen s001). It is not published, because it carries the owner's account name and network name. The controls quoted above are copied from it.

What would prove this wrong

By 31 December 2026 we will run a null agent, which opens the target app and stops, over all 80 tasks on two iPhones, the owner's and a freshly reset one. Our prediction: it passes at least the 7 Settings tasks above on the owner's phone and at least 6 on the reset phone (no storage banner). A second null agent that types the instruction's quoted query and stops will pass all 9 search tasks. If either passes fewer, our reading of the matcher or of the root screen is wrong. We also predict that once the twelve checks grade the navigation bar title, at least one of the six models drops a task it passes today.

Sources

• Li Hu, Kanghua Mo, Yingbin Jin, Qingqing Ye, Haibo Hu, "DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections", arXiv:2610.06898, 29 September 2026; code at anonymous.4open.science/r/DIBench-5432.
• Yeji Park, Jaeyun Shim, Taesik Gong, "Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces", arXiv:2610.07972, 6 October 2026.
• Blolabel, phoneshell: environments/, perception/tree.py, bench/verify.py, read 7 October 2026.
• Blolabel, phoneshell-bench, results/xv2.jsonl, six models, 9 September 2026.
• Blolabel, "Eighty iPhone tasks assert five things", on the seven tasks that check only the foreground app.