Eighty iPhone tasks assert five things, and seven of them only check which app is in front

Counted from the task files, not the docs: 153 assertions over 80 tasks, five kinds doing all the work, and four tasks whose checker grades the wrong leg of the instruction.

Our 80-task iPhone suite declares a checking vocabulary of twelve assertions and uses five of them. Seventy-six of the 80 tasks assert which app is in the foreground, and for seven tasks that is the only thing asserted at all. Three of those seven ask for a sequence of three app switches and then check one bundle id, which the first switch already satisfies.

What a task is, in this suite

A task is a YAML file in environments/. It has three parts and the third is the one that costs money to build: setup, deterministic steps that put the phone in a known state with no model involved; instruction, the single sentence handed to the agent; and checks, assertions run afterwards that decide pass or fail. The runner's own docstring states the contract: "The agent never sees the checks, never sees the setup script, and its own account of what it did counts for nothing."

Across the 80 files there are 269 setup and teardown steps and 153 checks. Every task caps the agent at 25 steps, every one of the 80. Instructions run from 21 to 116 characters, median 76. Difficulty is self-declared: 15 easy, 58 medium, 7 hard.

Five kinds do all the work

scripts/validate_tasks.py will accept twelve check kinds and phoneshell/bench/verify.py implements all twelve. The suite uses five. The other seven, including every one that reads a control's value rather than its presence, appear in zero tasks.

The same thinning shows in setup. Nine verbs are allowed, seven are used, and two of the seven appear once or twice: terminate 165 times, home 92, then clean_alarms 3, open_app 3, swipe 3, wait 2, tap 1. open_url and type are never used in setup, because the 20 probe tasks put the URL in the instruction and make reaching it part of the job.

Seven tasks a launch passes

Seven tasks have exactly one check, and in every case it is foreground_app. For weather.open ("Open the Weather app") that is the whole job and the check is exact. For three others it is not.

Checks run once, after the agent stops. There is no step-level assertion anywhere in the runner, so an intermediate state the instruction demands cannot be graded by construction. A task whose instruction has three legs and whose checker has one assertion is measuring the last leg and hoping for the first two. Two of the three sequencing tasks are worse than that, because the assertion names the app the sequence starts from.

Our judgement, not a measurement: these four tasks should carry a second assertion the shortcut cannot satisfy. Spotlight leaves a search field in the tree, and a detour through Notes leaves Notes in the app switcher. Both are readable. Neither is currently read.

What this costs in coverage

Deterministic checks only work where the phone will tell you the answer, and that is why the suite lives in first-party apps: 16 bundle ids, all Apple, with Settings (20 tasks), Safari (26, including the 20 probes), Clock (9) and Notes (5) carrying most of it. There is no Uber task, not because it is uninteresting but because nothing in a closed third-party app exposes the state a checker would have to read. That is the trade a benchmark makes when it refuses a model judge, and it is worth stating in public rather than in a footnote.

Twenty tasks do grade the model's words. answer_contains exists for the narrow case where the deliverable is information rather than a state change, and the code says so: for "what word is in that picture", the word is on screen either way, so every state check would pass without the model having read anything. Ground truth has to be fixed in advance, which is why those 20 probes run against pages we host rather than against content that can change.

Check it yourself

git clone https://github.com/Blomega/phoneshell && cd phoneshell

# every check kind, counted from the task files
python3 - <<'PY'
import yaml, glob, collections
c = collections.Counter()
for f in glob.glob('environments/*.yaml'):
    for chk in yaml.safe_load(open(f))['checks']:
        c[chk['kind']] += 1
print(len(glob.glob('environments/*.yaml')), 'tasks', sum(c.values()), 'checks', c.most_common())
PY

# the tasks whose only assertion is which app is in front
grep -L -e 'element_exists' -e 'regex_on_screen' -e 'answer_contains' environments/*.yaml

# the declared vocabulary, and the validator that enforces it
sed -n '20,30p' scripts/validate_tasks.py
python3 scripts/validate_tasks.py

One probe page is public if you want to see what a render-stratum task actually asks: blolabel.ai/probe/seven/. The word in that picture is absent from every accessibility node on the page.

What would prove this wrong

The claim is that the three sequencing tasks are satisfiable by their first leg. It is falsifiable in one run: add a second assertion to multi.app_switch and multi.return_after_detour that requires the detour app to be present, then re-run the same models. If pass rates on those two tasks do not fall for at least one model by 31 October 2026, the shortcut was never being taken and the tasks were fine as written. We will publish the paired numbers either way.