133 of 12,255 swipes were the iOS back gesture, and one tap in ten was a miss

iOSWorld's least-read appendix measures the two things iOS does differently from every other target, and both land on the rig rather than on the model.

Across 12,255 swipes by frontier models on an iOS agent benchmark, 133 were the left-edge rightward swipe that is iOS's only system-wide back gesture. That is 1.1%. On a platform with no hardware back button, the agents essentially never used the thing that replaces it. In the same trace, 10.26% of all vision-only taps were a miss followed by a retry within 60 pixels, 1,669 out of 16,267, and the paper reports that rate falling to about zero when the same tap goes through an XCUITest accessibility identifier instead of estimated coordinates.

Where these two numbers live

Appendix K of iOSWorld, arXiv:2606.09764, posted 8 June 2026. The appendix is titled "iOS-Specific Interaction Patterns" and it quantifies two things straight from the action trace of 133 tasks per model, rather than from pass rates. It is two paragraphs and one table in a 30-page paper, and it is the most directly actionable part of it for anyone building a rig.

The miss-rate proxy is stated precisely enough to re-implement: a tap_xy followed by another tap_xy within 60 px on the next step, computed over all 133 vision-only trajectories per model, taking the first tap in each step where a step emitted several. It is a proxy, not ground truth. A deliberate double-tap inside 60 px counts as a miss, and a miss that the model gave up on rather than retried does not count at all, so the number is neither a floor nor a ceiling. It is a consistent measure across models, which is what makes the comparison worth anything.

One tap in ten lands wrong and gets repeated

Per-model rows are Table 17 of arXiv:2606.09764. The pooled row is ours. Qwen3.5 35B-A3B is not in that table, so the pooled figure covers the five frontier runs only. Gemini's 5.3% comes with the paper's own caveat attached: its vision-only runs are shorter and leave 1,753 taps against 3,102 to 4,384 for the others, 10.8% of the pooled tap count, so the lower rate may be a shorter exposure rather than better aim.

A 10% miss tax is not a rounding error on a long-horizon task. The multi-app category in the same benchmark runs to 40 steps and beyond. If a tenth of the taps in a 40-step trajectory are wasted on a retry, four of the model's steps go to aim correction before any of them go to the task, against a 50-step budget.

The back gesture is the one iOS control agents will not learn

iOS gives you two ways back: an in-app chevron, whose position and label vary by app, or a rightward swipe starting at the left edge of the screen. There is no third. Out of 12,255 frontier-model swipes, 133 were that edge swipe, 1.09%.

The paper's reading, which we agree with, is that this is a gesture problem rather than a provider problem: the GPT models use it somewhat more, at 2.7% and 2.1% vision-only, but Claude and Gemini top out at 1.6% vision-only and 1.2% with the tree, and nothing is above 3%. Every family under-uses it by the same order of magnitude.

The mechanism is in the action translation table, and it is incomplete

iOSWorld's action space lists tap by accessibility identifier as a vision+XML action, alongside launch_app by bundle id, terminate_app and open_url. Its provider translation table then maps every provider's native click to tap_xy: Claude Computer Use left_click, OpenAI CUA click, Gemini CU click_at, Qwen click. No native action in that table maps to the identifier tap. launch_app appears only for Gemini, as open_app, and the action distribution bears that out: Gemini spends 18% of its actions on launch_app and every other model shows a dash.

So in the vision+XML setting the agent is handed a structured tree and, for four of six models, a click vocabulary that can only express a pixel. It reads an identifier and has to convert it back into an x and a y before it can act on it. The action distribution under vision+XML is 62% tap_xy for Opus and Sonnet, 55% for GPT-5.4, 37% for GPT-5.4 Mini, 54% for Gemini and 66% for Qwen3.5.

We could not close this one. The same appendix says the miss rate drops to about 0% "under XCUITest accessibility-ID taps", which implies identifier taps did happen, yet the action distribution table has no row for them and the translation table lists no provider action that produces one. Either identifier taps are folded into the tap_xy row, or the near-zero figure comes from a run not otherwise reported. The counts behind "about 0%" are not published. Treat that specific number as the authors' claim rather than as a measurement you can audit, and treat the 10.26% as the audited one.

What a rig should take from this

Both findings point the same way: the actions an iOS agent needs most are the ones a general computer-use vocabulary expresses worst. A pixel is a bad way to name a button, and a swipe is a bad way to say "go back". Both are fixable in the rig rather than in the model.

phoneshell resolves both as named operations. Actions.back() in phoneshell/actions.py opens with the comment "iOS has no back button", scans the current tree for a Button or Link whose text is one of back, cancel, done, close, the single left chevron or a bare less-than sign, taps it if present, and otherwise calls the gesture. The gesture, Gestures.back() in phoneshell/gestures.py, is edge_swipe("left", distance=0.6), which starts at x=0 at the vertical midpoint because the system gesture lives in the few points iOS reserves at the very edge and a swipe starting even slightly inboard is delivered to the app instead. A model driving this never has to know any of that; it emits "back".

The same applies to taps. Actions.tap_element() takes an element from the tree and taps it, so the coordinate is computed by the rig from the element's own frame rather than estimated by the model from an image. On the evidence above that is worth roughly a tenth of all taps, which is a larger effect than most prompt changes will buy.

This is not an argument that the model should be given fewer capabilities. It is an argument that the iOS-specific ones should be verbs in the action space, not gestures the model has to rediscover from a screenshot. The distinction shows up in the numbers: the models did have a swipe action with an optional origin, and still produced 133 back swipes out of 12,255.

Check it yourself

The appendix and both tables come out of the paper's HTML without a PDF reader:

curl -sL https://arxiv.org/html/2606.09764 -o iosworld.html
python3 - <<'PY'
import re, html
txt = html.unescape(re.sub(r'<[^>]+>', ' ', open('iosworld.html', encoding='utf-8').read()))
txt = re.sub(r'\s+', ' ', txt)
i = txt.find('Appendix K')
print(txt[i:i+2600])
PY

# the pooled rate
python3 -c "
t=[3102,3236,4384,3792,1753]; r=[314,329,541,392,93]
print(sum(r), '/', sum(t), '=', round(sum(r)/sum(t)*100, 2), '%')"

On the rig side, the two back paths are in this repository: grep -n "no back button" phoneshell/actions.py finds the label-first path, and grep -n "def back" phoneshell/gestures.py finds the edge swipe with its 0.6 distance. The 60 px threshold in the miss-rate definition is worth reproducing against your own traces: it is one pass over a step log, no device needed.

What would prove this wrong

The claim doing the work here is that the miss rate is about grounding rather than about the models being careless, and that identifier taps remove it. The paper asserts the second half without publishing counts, so it is the weak link.

Run the same near re-tap detector over vision+XML trajectories for a model that does have an identifier tap available, count both tap kinds separately, and publish the two rates. If identifier-routed taps show a near re-tap rate materially above zero, say above 3%, then the tree's coordinates are themselves stale or wrong often enough to matter and the advice to route taps through elements is weaker than stated. If the rate on coordinate taps in the same runs stays near 10%, the grounding story holds and the gap is the whole argument.

For the back gesture the falsifier is simpler and we will run it: give a model an explicit back action on the physical iPhone suite, count how often it is used against how often the trajectory needed to leave a screen, and report both. If a named back action goes unused at the same 1% rate, the problem is not the vocabulary and this note has the cause wrong. We will publish that count by 31 December 2026.