640 leftover alarms turned a 154 ms screen read into 24 seconds, and the tree came back 218 switches short

A benchmark task failed because of the residue of earlier runs of the same task. The mechanism is general: WebDriverAgent's cost per request grows with the accessibility tree, and past a few hundred rows the tree it returns is incomplete and says nothing.

On a physical iPhone 17 Pro Max on 8 September 2026, WebDriverAgent read the Clock timer screen in 154 ms, the alarm list in 2,612 ms, and the Add Alarm sheet over that list in 24,195 ms, a 6.1 MB response. The list held 640 alarms, 621 of them identical unlabelled artefacts saved by agents on earlier runs of a task that says "do not save it". At that size one read returned 631 cells and 413 switches with HTTP 200. There is one switch per alarm, so 218 were missing and nothing said so.

What happened, and what it broke

clock.alarm.set_time has a 240-second budget. At 24 seconds per observation it produced zero completed turns, then, before a guard was added, passed both of its checks from whatever the list behind the sheet happened to show: its pattern for 30 matched 10:30AM, Alarm. The task failed, and briefly passed, because of residue from earlier runs of itself. The measurement and the fixes are in FINDINGS.md sections 18 to 21, committed 8 September 2026 in the change that added a clean_alarms teardown to the three Clock tasks. The pre-registration for the next experiment then made a Clock tree read under 0.5 s a hard gate before any run could start.

For comparison, iOSWorld (arXiv:2606.09764, 8 June 2026) passes its agents a cleaned XCUITest tree capped at 200 elements and 15 levels. A cap like that keeps observation cost bounded. Applied to a 640-row list it would also hide most rows, by design rather than by accident, and an agent would have to scroll to see them.

The measurements

Depth is the other lever, and it has a knee. On a heavy third-party food list, raising snapshotMaxDepth added elements and time until depth 25, then only time.

Why a big tree is slow, and why it lies by omission

WebDriverAgent resolves a request by snapshotting the element hierarchy through XCTest, so a tap, a drag and a read each pay for the tree as it stands. That is our reading of the numbers above rather than something the runner documents: every operation we timed on the 640-row screen got slower together, by 4.2 times for a tap and 3.0 times for a drag. At the large end the snapshot also stops being complete, and it does so without an error, a truncation marker or a status change.

The asymmetry is the useful rule. A truncated tree can make something look gone. It cannot make something appear. Checks for presence survive a short read; checks for absence need the read to be settled, repeated or scrolled, and our element_exists check with negate: true now requires exactly that before it will score a pass.

What this changes if you run agents on real phones

Cleanup is throughput. A suite that mutates the device has to undo it, or its latency drifts upward run by run and the numbers drift with it. The alarm task did not get harder; its screen got 17 times more expensive to read.

Budget per observation, not per task. A fixed 240-second budget is ten observations at 24 s and more than a thousand at 154 ms. If observation cost is not logged per step, a timeout on a bloated screen reads as a model failure.

Set depth deliberately and escalate. Depth 20 read 59 of the 78 reachable elements in 0.21 s; depth 25 read all 78 in 1.27 s. phoneshell defaults to 20 and re-reads deeper only when the shallow read looks under-described. Drop visible and accessible from the snapshot unless a check needs them.

Never act twice on one read. Judgement from the delete incident: deciding each deletion from the row's own label and switch state, one delete per fresh read, was slower and correct; the batched plan was faster and removed an alarm belonging to the phone's owner.

Check it yourself

With bin/phoneshell up running from github.com/Blomega/phoneshell, open Clock on a screen of your choice and time a full read:

# time and size of one full tree read, session-free
curl -s -o /tmp/src.json -w '%{time_total}s %{size_download} bytes\n' \
  'http://127.0.0.1:8100/source?format=json'

# count switches and cells in that read
python3 -c "
import json
def walk(n):
    yield n
    for c in n.get('children') or []: yield from walk(c)
t = json.load(open('/tmp/src.json'))['value']
k = [n.get('type') for n in walk(t)]
print('cells', k.count('XCUIElementTypeCell'), 'switches', k.count('XCUIElementTypeSwitch'))"

# the same read without the two hit-tested attributes
curl -s -o /dev/null -w '%{time_total}s\n' \
  'http://127.0.0.1:8100/source?format=json&excluded_attributes=visible,accessible'

Depth is set per session through WebDriverAgent's settings endpoint, POST /session/<id>/appium/settings with {"settings":{"snapshotMaxDepth":25}}, which is what set_settings in phoneshell/wda/client.py sends. Creating that session sends the phone to the home screen, so set depth before opening the app you want to time. The cleanup that removes artefact alarms one verified row at a time is scripts/clean_alarms.py and Runner.remove_artifact_alarms. Reproducing the 640-row figures needs 640 alarms, and we do not recommend creating them on a phone you use.

What would prove this wrong

The claim is that WebDriverAgent's per-request cost scales with accessibility tree size, and that very large trees come back incomplete without signalling it. The prediction, with a date: before 31 March 2027, on the then-current WebDriverAgent release and a physical iPhone, a full /source read of a Clock alarm list with at least 600 rows will still take more than ten times the read of the timer screen on the same phone. A release that brings it within three times falsifies the scaling claim. A release that returns every switch on a 600-row list in five consecutive reads falsifies the truncation claim.

The limit: every number here comes from one phone, one app state and single reads rather than distributions. The depth table is one third-party app. Treat the ratios as measured and the precise milliseconds as one sample each.