Apple's onscreen awareness needs the app to declare a persistent identifier; phoneshell gets a MAC address and a raw pointer baked into one instead

WWDC26 session 343 shows a developer attaching a persistent EntityIdentifier to whatever is on screen so Siri can resolve "play the third one" without a word of pixel or tree parsing. That cooperation does not exist in the ordinary, unmodified consumer apps phoneshell drives, and its memory.py file quotes, from a real run, exactly what shows up instead: an identifier with a Bluetooth MAC address and a raw memory pointer baked into it, which has to be detected and stripped before two sightings of the same screen can be recognized as the same screen at all.

Apple's WWDC26 session 343, "Explore advanced App Intents features for Siri and Apple Intelligence," presented by Antonio Cancio of the App Intents team, covers onscreen awareness: the mechanism that lets Siri connect what is visible on a screen to entities and actions it understands. Without it, the session states, Siri has no way to resolve a reference like "play the third one" or know which artist is meant if the artist's name is not itself visible. The session's code examples show four ways a developer opts a screen into this: attaching an NSUserActivity with an appEntityIdentifier for a screen dedicated to one primary item; a per-view .appEntityIdentifier annotation for a view representing one entity among several; a collection annotation, .appEntityIdentifier(forSelectionType:), for lists, which fetches identifiers lazily and keeps working even after a selected row scrolls out of the view hierarchy; and UIKit and AppKit equivalents, AppEntityAnnotatable and UICollectionViewAppIntentsDataSource. Every one of these requires the identifier to be persistent, and the session separately stresses implementing fast display-representation queries, so Siri can resolve an on-screen entity without loading its full content, warning that without this Siri may ask for clarification or give up.

What Apple's onscreen awareness assumes the app will do

The design Apple describes puts the entire burden of stable identity on the app's own code. A developer who wants Siri to ground a reference to a song, a playlist, or a conversation must supply an EntityIdentifier that refers to the same underlying item every time it is shown, across launches, across scroll positions, across however the surrounding UI happens to be laid out that day. The session's sample code pairs a SongEntity or AlbumEntity type with a stable identifier string the app already uses internally, not with anything derived from the view hierarchy or the screen's rendered content. This works precisely because the app knows its own data model and is choosing, at build time, to expose a piece of it through a documented API. The mechanism has no fallback for an app that declines to participate: the session does not describe any way for Siri to infer an entity's identity from an app that has not adopted these annotations.

Source: Apple, WWDC26 session 343, "Explore advanced App Intents features for Siri and Apple Intelligence," presented by Antonio Cancio, read 30 October 2026.

What phoneshell gets from an app that never opted in

phoneshell drives ordinary, unmodified consumer iOS apps, none of which were built with onscreen awareness or any equivalent agent-facing API in mind. Its own grounding problem, deciding whether a screen the agent is looking at now is the same screen it saw before, is solved entirely from what WebDriverAgent's accessibility tree happens to expose, and memory.py's own comments document exactly how unreliable that is as a source of stable identity. The file quotes two real identifiers pulled from a live run of iOS Settings:

com.apple.settings.followUpGroupAsItem...group.account.61932825-AFFA-4F26-...
com.apple.settings.connectedHeadphone.38:C4:3A:26:96:EE-0x85a8cf0f0

The comment accompanying them is direct about what is wrong: the first contains a UUID, the second a Bluetooth MAC address and a raw memory pointer, both of which "survive within one launch and change on the next, which silently turns every revisit into a screen the agent has never seen." Where Apple's mechanism asks the app to hand over an identifier that means "this exact song" forever, iOS Settings, with no obligation to do any such thing, hands phoneshell an identifier that means "this exact Bluetooth session, at this exact address in memory, right now." memory.py answers this not by requesting better identifiers, which it has no way to do, but with a regex table, _ID_NOISE, that pattern-matches and strips UUIDs, MAC addresses, and similar instance-specific fragments before an identifier is allowed anywhere near a screen fingerprint.

What it means for grounding on iOS, with and without app cooperation

• Apple's onscreen awareness is not a general solution to the grounding problem phoneshell has: it only helps for the subset of apps whose developers adopt the annotations, and consumer apps adopt new App Intents APIs on their own schedule, typically slower than an agent needing to work on them today.
• phoneshell's regex-based noise stripping is a workaround for the absence of exactly the cooperation Apple's mechanism assumes, not a competing design: if every app phoneshell drove shipped onscreen awareness annotations, memory.py's fingerprinting problem would mostly disappear, because the app itself would be handing over the stable identifier phoneshell currently has to reconstruct by filtering out MAC addresses and pointers.
• The gap is a measure of how far consumer iOS apps are from agent-readiness: a MAC address and a raw pointer sitting inside what is nominally an accessibility identifier is the kind of implementation detail a developer would never expose on purpose to a system built to read it, which is exactly why phoneshell has to assume nothing about any identifier's stability until it is checked.

Check it yourself

# the real identifiers phoneshell quotes from a live iOS Settings run
grep -n "followUpGroupAsItem\|connectedHeadphone" phoneshell/memory.py

# the regex table that strips UUIDs, MAC addresses and similar noise
grep -n "_ID_NOISE" -A 10 phoneshell/memory.py

# confirm memory.py's match key is the screen fingerprint, computed after noise stripping
sed -n '1,24p' phoneshell/memory.py

# Apple's session page and its onscreen awareness chapter
open "https://developer.apple.com/videos/play/wwdc2026/343/"

# the specific annotation APIs named in the session
#   NSUserActivity.appEntityIdentifier, .appEntityIdentifier(forSelectionType:),
#   AppEntityAnnotatable, UICollectionViewAppIntentsDataSource

What would prove this wrong

We claim Apple's onscreen awareness APIs, as shown in WWDC26 session 343, require the hosting app to declare a persistent entity identifier at build time, with no mechanism shown for Siri to infer stable identity from an app that has not adopted them; that is checkable against the session itself and would be false if Apple's own documentation describes a fallback path we have not found. We also claim the two identifier strings quoted above are real output from a live run against iOS Settings, not constructed examples; that is checkable directly in memory.py's own comments, dated to this repository's history. We will revisit this piece if a future iOS release ships onscreen awareness support that works without app-side annotation, or if phoneshell's own identifier-noise table needs new patterns for an app we have not yet seen, by 30 April 2027.