DAC-Bench's branching tasks fall below 5% success past six steps; phoneshell's task schema has no branch to take

An ACL 2026 benchmark built 830 episodes around four task structures, one of them branching, and watched seven agents collapse on the branching ones. We read phoneshell's own task format to see where its 80 tasks would land. They would not land anywhere on that map: the format has no field for a branch at all.

DAC-Bench, published at ACL 2026 (pages 44578-44598 of the Long Papers volume) by Yuqing Zhang, Honghui Sheng, Xueyu Hu, Shengyu Zhang and Fei Wu, tests seven agents on 830 episodes and 11,345 action steps across 35 Android and iOS apps, built around four task structures: Sequential, Conjunctive, Conditional and Hierarchical. On the Conditional and Hierarchical tasks, success rate drops below 5% once a task reaches 6 to 8 steps, and branch accuracy, a new metric the paper introduces, averages 38% across all seven agents. phoneshell's own 80-task suite, read directly from environments/*.yaml, has no mechanism to pose a conditional or branching task at all. Every one of its tasks is Sequential in DAC-Bench's own taxonomy, because the schema that defines a phoneshell task has nowhere to put anything else.

Four structures, one cliff

DAC-Bench's stated complaint about prior mobile GUI benchmarks is that they mostly test "short, linear workflows and step-level accuracy," which rewards an agent for getting each individual tap right without ever checking whether it chose correctly between two valid paths. Its four task structures are designed to force that choice: Sequential tasks are a fixed order of steps with no decision point, Conjunctive tasks require satisfying multiple independent sub-goals, Conditional tasks require the agent to read the screen and pick between branches whose correct outcome depends on what it finds there, and Hierarchical tasks nest these inside one another. The paper's new scoring metric, a weighted longest common subsequence, is built to separate two different kinds of partial credit: how far an agent progressed along a correct branch (length-sensitive progress) from whether it picked the correct branch in the first place (branch accuracy). Averaged across seven agents, branch accuracy lands at 38%, and overall task success for tasks that run 6 to 8 steps falls under 5%, a result the authors attribute specifically to conditional decision-making rather than to step count alone.

That distinction matters for reading the number correctly. DAC-Bench's own data shows plenty of 6-to-8-step Sequential tasks that do not collapse to under 5%; the cliff belongs to tasks that also branch. A benchmark made entirely of long Sequential tasks would not be expected to hit that floor. Which is exactly phoneshell's situation: every one of its tasks is long enough to have steps in that range, but none of them branch.

What a phoneshell task can say

A phoneshell task is defined by three dataclasses in phoneshell/bench/schema.py: Step (one deterministic action, used for setup and teardown), Check (one assertion run after the agent finishes), and Task itself, which holds exactly one instruction string, one flat list of setup steps, and one flat list of checks. There is no field anywhere in any of the three for a branch, a condition, or an alternative outcome; the module's own docstring describes the whole format as three parts, run once, in order: "setup... instruct... checks." A real example, environments/clock_timer_set_hm.yaml, asks the agent to "set the timer to 1 hour and 30 minutes," with two checks that must both pass, which is Conjunctive at most in DAC-Bench's taxonomy and not Conditional: nothing about the task changes based on what the agent observes on screen. Searching every task file in the suite for anything resembling a decision point returns nothing: grep -lE "^(if|branch|condition)" environments/*.yaml matches zero of the 80 files, because the YAML format they are written in was never given the keys to express one.

What it means for anyone building a task set

• phoneshell's own step counts, 4.4 to 6.7 turns averaged per model, already sit near the bottom of DAC-Bench's 6-to-8-step danger zone, on tasks that never branch. Adding a single conditional fork to a task that already takes six turns would place it squarely inside the range where DAC-Bench measured a collapse, for a population of models it has not yet tried that fork on.
• phoneshell's Check objects already support multiple checks per task (DAC-Bench's Conjunctive structure), which is why the timer task above satisfies two independent assertions. Getting to Conditional would require a new field, something like a check or a setup step whose outcome determines which of two subsequent check lists applies, and nothing in schema.py today reads a screen state to pick between two paths before scoring.
• DAC-Bench's weighted longest common subsequence metric solves a scoring problem phoneshell does not currently have, because phoneshell's checks are binary pass/fail per assertion, not a measure of how far along a branch the agent got before choosing wrong. A phoneshell task set that added branching would need an equivalent partial-credit metric, not just a new YAML key.
• The 38% branch-accuracy average is itself the headline, not the 5% success floor: it says that even when branching tasks are scored generously for partial progress, picking the right branch is close to a coin flip across all seven agents DAC-Bench tested, none of which is a model phoneshell's own leaderboard currently runs.

Check it yourself

# confirm phoneshell has no branching or conditional structure
grep -n "class Task\|class Step\|class Check" -A10 phoneshell/bench/schema.py
grep -lE "^(if|branch|condition)" environments/*.yaml; echo "matches: $?"
ls environments/*.yaml | wc -l          # 80

# phoneshell's own step counts, for comparison against DAC-Bench's 6-8 step cliff
python3 -c "import json; d=json.load(open('site/results.json')); [print(k, v['avg_turns']) for k,v in d['board'].items()]"

# DAC-Bench's own numbers and code
curl -s https://aclanthology.org/2026.acl-long.2064.pdf -o /tmp/dacbench.pdf   # abstract and Table 1
# code and data: https://github.com/YuqingZhangMirror12/DAC-Bench

What would prove this wrong

We claim phoneshell's task schema cannot express a Conditional or Hierarchical task today. That is wrong the moment phoneshell/bench/schema.py gains a field that lets a check or setup step branch on an observed screen state, or the moment any task file encodes two alternative outcomes for the same instruction. We also claim phoneshell's current 80 tasks are all Sequential or, at most, Conjunctive in DAC-Bench's taxonomy. That is wrong if a close reading of any task file turns up a branch we missed; our own check was a plain grep for if, branch and condition across all 80 files on 15 October 2026, which is a necessary but not fully sufficient test for the absence of branching logic. We will re-run that check, and look for it by hand once more, if phoneshell's task count passes 120 or if a maintainer states that a branching task has been added, whichever comes first, and treat either as grounds to revisit this comparison.

Sources

• Zhang, Sheng, Hu, Zhang, Wu, DAC-Bench: A Decision-Aware Benchmark for Compositional Mobile GUI Tasks, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, pages 44578-44598, ACL 2026, read 15 October 2026.
• Code and data: github.com/YuqingZhangMirror12/DAC-Bench.
• Blolabel, phoneshell: phoneshell/bench/schema.py, environments/*.yaml (80 files), site/results.json, read 15 October 2026.