AmbiBench's non-interactive agents score zero on ambiguous instructions; phoneshell can't ask back

Two tested agents finish zero tasks on AmbiBench's most underspecified tier. phoneshell's 80 tasks all sit at the benchmark's most explicit tier instead, and its only confirmation gate checks for risk, never comprehension.

On AmbiBench's most underspecified instruction tier, two of seven tested mobile agents finish zero tasks: AutoGLM-9B and MobileAgentV2 (GPT-4o) both score a Task Success Rate of 0% on Ambiguous-level instructions (Fudan University, arXiv:2602.11750, submitted 12 February 2026). The paper's own metric for whether an agent even tries to close that gap by asking, Information Gain Rate, is 2.4% for Qwen-3-VL-8B. phoneshell cannot be placed anywhere on this scale: every one of its 80 tasks is written at AmbiBench's most explicit tier, and nothing in its tool set lets an agent ask a clarifying question at all.

What AmbiBench measured, and when

AmbiBench organises 240 ecologically valid tasks across 25 applications into four clarity tiers grounded in Cognitive Gap theory: Detailed, Standard, Incomplete, and Ambiguous. Its evaluator, MUSE, is an MLLM-as-judge system scoring three dimensions: Outcome (Requirement Coverage Rate, Task Success Rate), Execution (Key Step Hit Rate, Action Redundancy Rate, Error Termination Rate), and Interaction (Dialogue Compliance Rate, Information Gain Rate, the share of the task's information gap an agent's questions actually close). Seven agents were tested: UI-TARS-7B, AutoGLM-9B, Qwen-3-VL-8B, AppAgent and MobileAgentV2 (both GPT-4o), and two configurations of Fairy (GPT-4o paired with Gemini-3-Flash or with UI-TARS-7B).

These are averages across all four tiers (Table 2 of the paper). The level-specific numbers are starker: AutoGLM reaches a 65.2% Task Success Rate on Detailed instructions but a 0% Task Success Rate on Ambiguous ones, with no interaction in between to recover. A mini-ablation on Fairy, restricted to Incomplete-tier tasks it had already solved, turned its interaction off: Task Success Rate fell from 100% to 0% and Requirement Coverage Rate fell to 23.8%. The paper's reading is that non-interactive agents do not fail by stopping, they fail by guessing: several keep a respectable Key Step Hit Rate on Ambiguous tasks while Task Success Rate sits at zero, meaning they execute a plausible-looking path toward the wrong, invented requirement.

phoneshell has a confirmation gate. It is not a clarification gate

phoneshell does have a mechanism that pauses and reports back to a human mid-task: phone_tap and phone_macro_run both accept a confirmed flag, and refuse to proceed the first time without it. But what triggers that refusal is Guard.classify_tap (safety.py:33), which runs the tapped element's visible label text through a list of regex patterns for words like "delete," "send," "pay," or "order," and returns needs_confirmation only when one matches. It has no concept of whether the agent correctly understood the instruction. A model that misread "collect parcel" as "collect package" and typed the wrong text would sail through this gate, because nothing about that action looks irreversible. AmbiBench's Dialogue Compliance Rate and Information Gain Rate measure a different event entirely: a model recognising that it does not have enough information and generating a genuine question, rather than guessing. grep -rn "clarify\|ask_user\|needs_clarification" phoneshell/ returns nothing, because that verb does not exist in the tool set at all, under any name.

What it means for anyone benchmarking a phone agent

• A high pass rate on a Detailed-only task suite says nothing about what happens when an instruction is merely Standard, let alone Incomplete or Ambiguous. AmbiBench's own data shows the drop is not gradual for several agents: Task Success Rate can hold in the 60s on Detailed and hit zero on Ambiguous with nothing in between.
• A safety confirmation gate and a comprehension clarification gate are not the same mechanism and do not substitute for each other. phoneshell has shipped the first; the paper is measuring agents that at least attempt the second.
• If you want phoneshell's leaderboard to say anything about the Ambiguous tier, the task files need instructions that genuinely underspecify something (which contact, which of three identically-named reminders, which of two installed timer apps), and the tool set needs a verb an agent can call to ask back, with a needs_confirmation-style refusal path distinct from the safety one.

Check it yourself

• The paper's Table 2 and level-specific Task Success Rate figures: https://arxiv.org/abs/2602.11750, Fudan University, submitted 12 February 2026.
• No ambiguity-resolution verb exists: grep -rn "clarify\|ask_user\|needs_clarification" phoneshell/ returns nothing.
• The confirmation gate is risk-only: sed -n '1,40p' phoneshell/safety.py, specifically classify_tap's regex match against cfg.safety.confirm_patterns.
• Instruction completeness: grep "^instruction:" environments/*.yaml across all 80 files; every one names the app, the target value, and the end state explicitly.

What would prove this wrong

The claim is that phoneshell's agent will never ask a clarifying question because it has no way to, not because its current tasks never need one. Test: write five new tasks that genuinely underspecify something a human would ask about (for example, three identically-titled reminders, "delete the one about the meeting"), and run them. If the agent asks a sensible clarifying question through any existing channel, that channel exists and this note is wrong about the tool set. We expect it to guess, pick one, and report success regardless of which one it picked. We will publish the five tasks and the transcripts, or say we did not run it, by 15 November 2026.