Six models, 58 tasks on one iPhone, and not a single pair separates

Adding 34 tasks to the benchmark took the ranking further out of reach, not closer. The reason is arithmetic that contains no sample size: McNemar needs six one-directional disagreements to reach p < 0.05, and our widest pair has had four since the first run.

Six frontier models from six vendors ran the same 58 tasks on a physical iPhone 14 Pro on 9 September 2026. They scored between 89.7% and 96.6%, and under McNemar's exact test not one of the fifteen model pairs separates at p < 0.05. The suite had been 24 tasks. Going to 58 did not fix the ranking, it moved it further out of reach: the widest gap on the board went from needing about 33 tasks to needing about 80.

What we ran, and when

The first board ran six models over 24 tasks on a physical iPhone for $4.29 of API spend: Kimi K3 at 91.7%, GPT-5.1, Claude Opus 4.8 and Qwen3-Max all at 83.3%, Gemini 3.1 Pro at 79.2%, DeepSeek v3.2 at 75.0%. Three identical numbers sitting in the middle of a sorted list is the tell. Sorting is not ranking, and a page that prints six percentages in descending order reads as an ordering whether or not the data contains one.

The sweep meant to fix it ran on 9 September 2026: 58 tasks attempted by all six models, round-robin by task rather than blocked by model so that an interrupted run still leaves a balanced paired design, $9.25 and about five hours of wall clock on the phone. The 20 probe.* tasks were excluded, because they are a vision ablation's stimuli, built so a model with eyes gets all ten and a model without gets none. Tasks a model was never asked, because of a provider rate limit or a missing app, are recorded as not scored rather than as failures, so they never enter a denominator.

The same saturation is visible one platform over, and it is better documented there. AndroidWorld shipped in May 2024 with 116 programmatic tasks across 20 Android apps and a best agent at 30.6%. By 8 October 2025 AGI Inc was claiming 97.4% on it, in a post with no confidence interval, no run count and no per-task results. MobileWorld, posted to arXiv on 22 December 2025 and published at ACL 2026, opens by stating that AndroidWorld agents "achieving over 90% success rates indicate its saturation", then rebuilds the problem with 201 tasks where the best agentic framework reaches 51.7% and the best end-to-end model 20.9%.

The board, and the interval the board did not print

Every score below is a Wilson 95% interval computed from the same paired run, not a measurement. On 24 items the narrowest interval is 23.5 points wide and the widest is 32.9. All six overlap each other.

Fifteen pairs, zero separations, at both sizes

Models run on the same tasks are a paired design, so comparing two percentages as independent proportions throws away the only information that makes a small sample decidable: which tasks the two models disagreed on. McNemar's exact test uses exactly that and nothing else. Run over the 24-task board it returns this.

Three pairs are marked never. Their disagreements are exactly symmetric: each model wins precisely the tasks the other loses, so scaling the suite cannot separate them because there is nothing to scale. To this instrument those three models are the same model. On the 58-task sweep two pairs are still exactly symmetric, Claude Opus 4.8 against GPT-5.1 and DeepSeek v3.2 against Gemini 3.1 Pro.

Six disagreements is the floor, and it does not move with sample size

With all disagreements in one direction, McNemar's exact two-sided p is 2 x 0.5^b, where b is the count of tasks one model passed and the other failed. That expression contains no n. Five one-directional disagreements give p = 0.0625 and six give p = 0.03125, so six is the smallest number of disagreements that can reach p < 0.05, on a suite of 24 tasks or a suite of a million.

Where the information went when the suite got bigger

A task every model passes and a task no model passes both carry zero information about ranking. On the 24-task board, 12 tasks were passed by every model and 2 by none, leaving an instrument 10 items wide behind a 24-item facade. On the 58-task sweep, 46 tasks were passed by every model. Eleven discriminate. One is failed by all six, which follows from the other two counts. So 2.4 times the tasks bought exactly one additional discriminating item, and the share of the suite at ceiling rose from 50% to 79%.

That inversion is the whole mechanism. McNemar sees only disagreements, so adding tasks helps only if the new tasks provoke them. These did not. The measured disagreement rate fell from about 17% to about 5%, and because required sample size scales inversely with that rate, the requirement rose from about 33 tasks to about 80 while the actual sample grew from 24 to 58. The earlier projection that 60 tasks would settle the board was a power calculation performed on tasks that had not been written yet, and it was only as good as the assumption that they would resemble the ones already run. They did not.

What this changes if you build or buy one of these

If you publish a leaderboard, publish b and c. A sorted column of percentages is not a ranking and cannot be checked by a reader. An AndroidWorld leaderboard aggregator, snapshot dated 8 September 2026, prints scores to one decimal place, Qwen3.8 Max at 85.3% over Qwen3.8-Flash-Next at 84.5%, with no interval, no task count and no per-task outcomes. That makes an independent paired test impossible even for a reader who wants to run one. AGI Inc's 97.4% claim from 8 October 2025 has the same shape.

If you are choosing a model to drive Apple's own apps, the accuracy axis is close to spent. Six models from six vendors land between 89.7% and 96.6% on this suite and are statistically indistinguishable, while price across the same six moves by a factor of about twenty-three: Claude Opus 4.8 cost $0.1130 per task against Qwen3-Max's $0.0076 for the same 83.3%. Judgement, not measurement: on first-party iOS the decision is cost and latency, and the accuracy column is close to decoration.

If you are writing tasks, write tasks that fail. The eleven live items point at where they are: multi-column picker wheels, cross-app creation flows, scrolling to a true list end, long typing, and recovering from a wrong starting state. The other lead is third-party apps, where the accessibility tree stops describing the screen. Our own trace of that has a chat app returning WAMessageBubbleTableViewCell where a button name should be. MobileWorld took the same route on Android and got the best framework down from over 90% to 51.7% by making tasks long-horizon and cross-app: 27.8 completion steps on average against AndroidWorld's 14.3, and 62.2% multi-app tasks against 9.5%.

And audit failures before believing them. Epoch AI's 30 October 2025 review of OSWorld found roughly 10% of tasks contain serious errors that make them invalid, about 10% depend on live internet data so their difficulty drifts, and around 10% of instructions changed even after the July 2025 revision. Our own first honest run scored 85.4% of attempted tasks, and reading every failure against what the agent actually said showed four of six were our fault, not the model's.

Check it yourself

The arithmetic in this note needs no phone. The threshold claim is one line:

python3 -c "import math; print([round(2*0.5**b, 5) for b in range(1,9)])"
# [1.0, 0.5, 0.25, 0.125, 0.0625, 0.03125, 0.01563, 0.00781]

The test, the intervals and the power calculation live in one module, phoneshell/bench/stats.py, as wilson, mcnemar_exact, min_discordant_for_significance and required_n. Both published required-sample figures reproduce from a 4-0 split, and no other split up to b, c = 11 reproduces either, apart from the same split with the arms swapped:

from phoneshell.bench.stats import wilson, mcnemar_exact, required_n

wilson(20, 24)          # 64.1% to 93.3%
mcnemar_exact(4, 0)     # 0.125
required_n(4, 0, 24)    # 33     the 24-task board
required_n(4, 0, 58)    # 80     the 58-task sweep
required_n(2, 2, 24)    # None   exactly symmetric, no suite size separates them

To regenerate the paired analysis from a results file, including Wilson intervals, all fifteen pairwise tests and the item statistics that produce the 46 / 1 / 11 split:

./.venv/bin/python scripts/analyse_models.py xv2

To run the sweep itself, round-robin by task and resumable per (model, task):

./.venv/bin/python scripts/run_sweep.py --run xv2

The task definitions are one YAML file each under environments/, so environments/clock_timer_set.yaml and environments/resilience_wrong_app.yaml are two of the eleven items above. The 16 held-out tasks are not in the repo, on the principle that a benchmark whose entire answer key is public becomes training data. Per-task outcomes are published at blolabel.ai and the task set is mirrored at huggingface.co/datasets/blolabel/phoneshell-bench.

What would prove this wrong

The claim is that a benchmark built from Apple's own applications cannot separate frontier models at any suite size, because those apps are at ceiling rather than because the sample is small. The prediction, with a date: by 31 March 2027, no pair of the six models on this board will reach six one-directional disagreements on a first-party iOS task set, however large it grows. One pair reaching b >= 6 on first-party apps alone falsifies it, and we will publish that as a correction if it happens.

The second prediction is directional and cheaper to test. A stratum of third-party apps, where control labels are class names rather than descriptions, will produce a disagreement rate above the 5% measured here on the first sweep it runs. If it does not, then the ceiling is a property of frontier models rather than of Apple's accessibility tree, and the argument in this note is about the wrong thing.

One caveat that limits everything above. This is one instrument, one phone, one day. Six models on 58 tasks is 348 cells, which is a small experiment by any standard other than the cost of running it on real hardware, and the sweep ran on a physical iPhone 14 Pro rather than the iPhone 17 Pro Max that produced the rest of this repo's numbers. Treat the ceiling as measured and the projection past 80 tasks as an assumption, since the last projection of that kind was wrong by a factor of two and a half.