OCR read thirteen iPhone screens and the accessibility tree already had everything on eleven of them

Vision OCR against the accessibility tree, one frame at a time, on eleven of thirteen first-party screens the image has nothing to add. The two exceptions are map tiles and book covers, which is to say text baked into pixels.

We read thirteen first-party iPhone screens twice, once through the iOS accessibility tree and once through macOS Vision OCR on the screenshot of the same frame, and diffed the words. On eleven of them, every word OCR could read was already in the tree. Two screens carried text that exists only as pixels, and both are the same phenomenon: characters rendered into a raster image. Maps, where the words are printed on map tiles, and Books, where they are printed on cover artwork. In Books, author, bestselling and an author's surname appear nowhere among the screen's 521 raw accessibility nodes.

Why we measured coverage before writing another vision task

Our first screenshot ablation (phoneshell FINDINGS.md section 23) reported that the image was worth zero points. It was run with a 235x512 thumbnail on 24 tasks against Apple's own apps, and a thumbnail that small is not a test of vision. Before re-running it we asked a cheaper question: on an iOS screen, is there anything a model could only learn by looking? That is measurable rather than arguable. Whatever OCR reads and the tree lacks is render-only content by definition, and a task whose answer lives there is the only kind of task that can separate the two conditions.

The survey ran on a physical iPhone 17 Pro Max on iOS 26.6, driven from macOS 26.5, with scripts/render_gap.py in the phoneshell repository. OCR is Apple's own Vision framework running locally on the Mac through pyobjc, so nothing leaves the machine and no screenshot is written to disk. The result is recorded in FINDINGS.md section 25 and published in the 15 September 2026 commit.

It points the opposite way to the headline number in iOSWorld (arXiv:2606.09764, 8 June 2026), which reports that adding a cleaned accessibility tree to a screenshot improves frontier models by up to 26 points on its 133 tasks. That is not a contradiction. iOSWorld runs 26 purpose-built apps in the iOS simulator, and its tree is filtered to interactive, visible elements and capped at 200 elements and 15 levels. We are measuring Apple's shipping apps, where the tree is unusually complete, and we are measuring the gap in the other direction: what the picture adds once the tree is already there.

Thirteen screens, two with anything to see

Coverage is the share of OCR-read words on the screen that the accessibility tree also carries. Render-only is what survives three noise filters, described below. The eleven screens with a recorded coverage figure are listed; two of the thirteen were surveyed without a figure being written into the note, and we are not going to reconstruct one from memory.

Files is the row that shows why a raw diff is not evidence. Its coverage is 20% and its render-only count is zero: OCR chopped file names and sizes differently from the way the tree reports them, so almost every word looked new and none of it was. Read the coverage column as a measure of how the two instruments tokenise, and the render-only column as the finding.

How a survey like this manufactures its own answer

The first pass reported render-only content on all seven screens it covered, and nearly all of it was noise. OCR returned ecuri, rivacy and oca, which are fragments of Security, Privacy and Location, all three of which the tree has. It returned 29h, 39m and 29h41m from a Live Activity in the Dynamic Island. It returned ac for the Calculator key the tree labels All Clear, and 115kb and 217kb for file sizes the tree carries with different spacing.

Left unfiltered, that produces a vision stratum built entirely from artifacts, which is the same mistake as the thumbnail, wearing different clothes. Three filters remove it: drop tokens under four characters, drop anything matching the live-activity pattern ^(\d+[hms])+$, and drop any word that is a substring of a longer word the tree already has. Both survivors, Maps and Books, were then checked directly against the raw tree rather than trusted from the diff.

What this changes for an agent that drives an iPhone

Send the screenshot when the target is rendered content, and not otherwise. On a screen like Calculator or Clock the image is a token tax with no measurable return: our pre-registered ablation measured it at 35% extra cost per task on render-dependent pages and 43% on tree-sufficient ones, with steps unchanged at 2.0 to 2.3 either way. On a map tile or a book cover it is not marginal, it is the task: 30 of 30 with the image, 0 of 30 without, McNemar exact p < 0.0001.

That makes it a per-screen decision rather than a global setting, and the signal is already computable. The same diff that produced this table is the switch: if the filtered gap is empty, the tree is the screen.

There is a second reading worth stating, because it is the same set of pixels from another direction. Text baked into an image has been a documented accessibility failure since WCAG 2.0 success criterion 1.4.5, Images of Text. The words our agent cannot reach on a Books cover are the words a screen reader cannot announce either. Work like the A11y-CUA dataset (arXiv:2602.09310, CHI 2026) and the MCP accessibility-tree proposal (arXiv:2608.24898, 13 July 2026) is arguing that agents and assistive technology should read interfaces through the same semantic layer. On eleven of thirteen Apple screens, that layer is already complete enough to do the whole job.

Check it yourself

The script is 160 lines and the whole method is in it:

git clone https://github.com/Blomega/phoneshell && cd phoneshell
bin/phoneshell doctor        # names every missing piece
bin/phoneshell up            # brings the bridge up over USB

./.venv/bin/python scripts/render_gap.py Settings Clock Calculator Safari Maps Books

Each screen prints how many words OCR read, how many of them the tree has, the raw gap, and what survives the filters, then the run ends with a verdict line. The filters are the is_fragment, LIVE_ACTIVITY and len(w) >= 4 conditions in gap_for_screen. OCR is phoneshell/perception/ocr.py, which calls the macOS Vision framework through pyobjc and returns "inconclusive" rather than a wrong answer when the bindings are absent. Pass --private to run it over third-party apps: it prints counts only and never a word, so the statistic can be published while the content stays on the phone. The script also refuses to measure a screen whose app never came to the foreground, because open_app reports success either way and the home screen scores about 46%.

The tasks built on this survey are the probe.* set in the public dataset at huggingface.co/datasets/blolabel/phoneshell-bench.

What would prove this wrong

The claim is narrow: on Apple's shipping apps, the accessibility tree describes the screen so completely that a screenshot adds nothing on eleven of the thirteen screens surveyed, and the two exceptions are both text rendered into a raster. It fails if any of these happens.

• Someone re-runs render_gap.py on the same thirteen screens under a later iOS 26 point release before 31 December 2026 and finds a third screen with five or more render-only words surviving the filters. Our prediction is that they will not.
• A first-party screen turns out to hide a control's state in pixels rather than in a value or a checkmark element. We found one case of the second kind already: the chosen row in Auto-Lock is identified only by a Button labelled checkmark, and only in the raw tree.
• The corollary fails the other way: a survey of thirty third-party apps returns tree coverage above 90%, which would mean the vision case we are reserving for other developers' apps is not there either.

The strongest caveat is sample size. This is one read of each screen on one device on one OS build, by one instrument. Coverage is a property of a screen's current contents, so a Maps screen with the search sheet open and a Maps screen showing a route are two different measurements.