Fourteen overlay shapes, and the ordered moves that close each one

A sheet is recognised by starting part-way down the screen, not by being large, which is why the iOS share sheet at y=478 was invisible to a size-based detector. The reflex lives in the harness because instructing a model to ignore popups has been measured and does not work.

Almost everything an app puts in front of a phone agent is one of fourteen shapes, and each shape has a known way out. This is the catalogue our harness works through, published as data: what each overlay looks like in the accessibility tree, the ordered moves that close it, and the two things that must never be tapped. The reason it lives in the harness rather than in the prompt is that telling a model to ignore popups does not work, and there is a measurement for that.

The catalogue

The ordering inside each row matters: it runs from the move that is most certain and least destructive to the one most likely to have a side effect. Tapping a labelled "Not now" beats guessing at a backdrop, which beats a back swipe that might also leave the screen you wanted.

Source: phoneshell/agent/playbook.py, which is the file the harness reads and also the text handed to the model when the reflex fails. Fourteen entries, each with a tell, an ordered moves list, and an optional never.

A sheet is recognised by where it starts, not by how big it is

The detector that looks for the largest container picks the app's own full-screen window every time. That is how the iOS share sheet stayed invisible to ours: it is a Popover whose top edge sits at y=478, well inside a window that starts at zero and is larger than it. The rule that works is geometric and boring. A sheet starts part-way down the screen and reaches the bottom. Size is not the signal, offset is.

Three related gesture facts fall out of the same geometry, and each was a real bug. A drag through the middle of the screen pans the map underneath instead of scrolling the results sheet on top of it, which reads to an agent as "scrolling does nothing", so the drag has to start inside the container that actually scrolls, found in the raw tree. A page flip needs momentum, because a drag that ends stationary snaps back to the page it started on. And a top-edge gesture must start at exactly y=0: from there the top-left swipe opens Notification Centre and the top-right opens Control Centre, repeatably. Whether y=1 also works is untested.

Why the reflex sits in code rather than in the prompt

One alternative is to describe the popups in the system prompt and let the model deal with them. That has been measured, and it does not work. "Attacking Vision-Language Computer Agents via Pop-ups" (arXiv:2411.02391) puts adversarial popups into OSWorld and VisualWebArena and reports an 86% attack success rate, meaning the frequency with which agents click the popup, along with a 47% drop in task success. The defenses tested are the ones you would reach for first: instructing the agent to ignore popups in the system prompt, and adding an advertisement notice. Neither reduced the attack success rate by more than 25%.

That is an adversarial setting, and ordinary commercial popups are not adversarial. The relevant part transfers anyway: a model asked to reason about an overlay in the same context where it is reasoning about the task will attend to the overlay, and the overlay is designed by someone whose job is to capture attention. A deterministic dismissal in the harness does not have attention to capture.

The cost argument is the same shape. An agent that has to reason about a promo sheet burns its budget narrating it, and turns are the expensive unit on a phone, not milliseconds. Fourteen ordered lists of moves cost nothing per step and remove that reasoning entirely.

What to take from this if you are building one

Write your own version of this table before you write dismissal logic, because the table is the part that survives. Our moves reference our gesture layer, but the tells are properties of iOS: a Popover is a Popover, an action sheet ends in Cancel, a paywall contains the word "month".

Keep the never-tap list separate from the moves. It is the only part with a consequence that cannot be undone by observing again, and it is the part a model should not be allowed to override with reasoning, however good the reasoning sounds.

Last, make the stability gate tolerate motion rather than demand stillness, or overlays will cost you far more than they should. Waiting for two byte-identical frames never converges on a screen with a countdown timer: a promotional sheet reading "40% off flash deals 14:19" made every observation pay the full timeout and turned a 2 second step into a 50 second one. The gate now settles when under 1.2% of pixels differ, because a clock moves well under 1% of the screen and a real transition moves tens of percent.

Check it yourself

The catalogue is one Python file with no dependencies, so it can be read, printed and diffed without a phone attached.

git clone https://github.com/Blomega/phoneshell && cd phoneshell

# the catalogue as the model receives it
python3 -c "from phoneshell.agent import playbook; print(playbook.as_text())"

# the fourteen entries, their tells and their never-tap rules
grep -n "kind=" phoneshell/agent/playbook.py
grep -n "never=" phoneshell/agent/playbook.py

# reproduce the share sheet case: open Safari, tap share, then dump the raw tree
bin/phoneshell up
curl -s "localhost:8100/source?format=json" | python3 -c "
import json,sys
def walk(n,d=0):
    r=n.get('rect') or {}
    if n.get('type')in('Popover','Sheet','Other') and (r.get('y') or 0)>100:
        print(n.get('type'), 'y=',r.get('y'), 'h=',r.get('height'), n.get('label') or '')
    for c in n.get('children') or []: walk(c,d+1)
walk(json.load(sys.stdin)['value'])"

The last command prints the candidate overlays by vertical offset. On a screen with the share sheet open, the row you want is the one that starts well below zero and runs to the bottom, not the tallest one.

What would prove this wrong

• Find a fifteenth shape that none of these fourteen tells matches, on a top-100 iOS app, and the claim that the catalogue is near-complete for consumer apps is wrong. We expect to be told about at least one: our prediction is that the catalogue needs fewer than five additions to cover the top 100 free iOS apps in the US App Store, and we would revise it on the sixth, through 30 June 2027.
• Measure a deterministic dismissal reflex against a model instructed to dismiss overlays itself, on the same task set, and find no difference in turns or in success. That would undercut the reason this is in the harness at all. We have not run that ablation; the popup-attack result is evidence about adversarial overlays, not about ordinary ones, and we are borrowing it.
• The number we are least sure of is y=478 for the share sheet. It is one observation on one device on iOS 26.6, and the geometry will differ by device and by OS version. The rule that matters is "starts part-way down", not the specific value.