DAC-Bench puts a number on compositional collapse: branch accuracy averages 38%, and three of phoneshell's 80 tasks share the structure that falls hardest

DAC-Bench (ACL 2026) scores seven agents across 830 episodes and four compositional task structures: success falls below 5% on 6 to 8 step tasks and branch accuracy averages 38%. phoneshell's own suite has exactly three tasks built from the Sequential structure that taxonomy names, and we already know its checkers grade them with one assertion that cannot see a branch at all.

DAC-Bench, published at ACL 2026, evaluates seven agents on 830 episodes and 11,345 action steps across 35 apps on Android and iOS, sorted into four task structures: Sequential, Conjunctive, Conditional, and Hierarchical. Across all seven agents, success falls below 5% on tasks needing 6 to 8 steps, and a new weighted longest-common-subsequence metric puts average branch accuracy at 38%. phoneshell's own 80-task iPhone suite has exactly three tasks that fit the Sequential structure by name, and this site already showed, on 20 September 2026, that each one is satisfiable by its first leg because its checker asserts only the final app in front.

Four structures, one metric built to catch a shortcut

DAC-Bench's premise, per its abstract, is that existing mobile GUI benchmarks measure short, linear workflows and step-level accuracy, which says little about long-horizon planning or branching decisions. Its four structures separate different ways a task can require more than one path: Sequential (do A, then B, then C, in order), Conjunctive (multiple conditions must all hold), Conditional (the path depends on what the phone shows), and Hierarchical (a task built from sub-tasks). To grade partial credit without rewarding a lucky shortcut, the authors add a weighted longest-common-subsequence metric for "length-sensitive progress and branch correctness." The published abstract does not give the metric's formula or a per-structure breakdown; those numbers are not reported here.

Three tasks, one taxonomy, one shared weakness

We already documented, task by task, that multi.app_switch ("Open the Calculator, then open Safari, then go back to the Calculator"), multi.return_after_detour, and multi.three_apps are the only phoneshell tasks whose instruction names an ordered sequence of more than one app, and that each one's sole check is foreground_app against the last app named. Two of the three name the sequence's starting app as that final assertion, which a model satisfies by opening it and stopping. DAC-Bench's taxonomy gives that exact shape a name: Sequential. Its finding, independently, is that this is the shape of task every one of seven agents is worst at once step counts rise, and that a checker scoring only the endpoint cannot tell a completed sequence from an abandoned one, which is the branch-correctness gap its weighted-LCS metric exists to catch.

What it means for someone building this kind of suite

Our judgement, not DAC-Bench's: a benchmark dominated by single-leg, final-state checks is not wrong to exist, but it measures a narrower claim than "this agent can operate a phone." DAC-Bench's seven-agent, 830-episode result says the collapse phoneshell's three sequencing tasks hint at is not an artefact of a small suite: it holds at scale, across two platforms, with a metric purpose-built to catch exactly the shortcut phoneshell's own checker allows. The useful next step for phoneshell is narrow and already known, from the earlier piece on this site: give multi.app_switch and multi.return_after_detour a second assertion the shortcut cannot satisfy, which is the same fix this site proposed on 20 September 2026, independent of DAC-Bench.

Check it yourself

git clone https://github.com/Blomega/phoneshell && cd phoneshell

# the three tasks that match DAC-Bench's Sequential structure
grep -l "then open\|then go back\|then the\|then come back\|End on" environments/*.yaml

# confirm each one's check count and kind
python3 - <<'PY'
import yaml, glob
for f in ('environments/multi.app_switch.yaml', 'environments/multi.return_after_detour.yaml',
          'environments/multi.three_apps.yaml'):
    t = yaml.safe_load(open(f))
    print(f, len(t['checks']), [c['kind'] for c in t['checks']])
PY

DAC-Bench's own repository, github.com/YuqingZhangMirror12/DAC-Bench, held no README content beyond its name as of 20 October 2026. The per-structure and per-agent tables that would let us check this more precisely are not reported here because we could not find them published anywhere we could cite.

What would prove this wrong

If DAC-Bench's full paper or appendix publishes a per-structure success table showing Sequential tasks do not collapse worse than Conjunctive, Conditional, or Hierarchical ones at the same step count, the comparison drawn here would need revision. If phoneshell adds a branch-aware second check to its three sequencing tasks and a re-run shows no model's pass rate on them falls, our 20 September prediction about those three tasks, not DAC-Bench's broader finding, would be wrong. We will look for the full paper again by 17 November 2026.