We gave three frontier models the same task on a real iPhone, once with a screenshot and once without, and varied only where the answer lived. When it was in the accessibility tree the picture changed nothing. When it existed only as pixels, the picture was the entire task.
See the result →Every task was run twice by the same model, back to back, with the order alternating. The only thing that differed between the two arms is whether the model was shown the screen.
| Where the answer lives | With the image | Tree only | McNemar exact |
|---|---|---|---|
| Only in the pixels a word drawn into a picture, no alt text |
30/30 100% |
0/30 0% |
p < 0.0001 |
| In the accessibility tree the control: same page, same question |
29/30 96% |
30/30 100% |
p = 1.0 |
The interaction is the finding, not the headline number. The control row shows nothing at all. Same pages, same navigation, same question shape, same resolution, same prompt: the only difference is where the answer sits. So the effect cannot be an artifact of the harness, and a model that scores 100% on one row scores zero on the other.
| Model | With the image | Tree only | p |
|---|---|---|---|
| Gemini 3.1 Pro | 10/10 | 0/10 | 0.0020 |
| Kimi K3 | 10/10 | 0/10 | 0.0020 |
| GPT-5.1 | 10/10 | 0/10 | 0.0020 |
Carrying the screenshot is a pure token tax. Step counts were unchanged either way, so the model does not work harder with it, only more expensively.
Not as a global setting. On a labelled screen the image buys nothing; on a map tile, a chart, a book cover or a canvas it is the whole task.
Across thirteen first-party screens, eleven were fully described by their tree. Third-party
apps are a different story: one chat app returns
WAMessageBubbleTableViewCell where a button name should be.
An earlier version of this experiment reported that vision was worth nothing, and it was wrong. The image had been downscaled to 235×512, so the vision arm was never a test of vision. That retraction is why the method below exists.
The design, the primary endpoint, the minimum detectable effect and the falsification condition were committed to git before the run. The sequence is auditable.
Ten pages, each with one word in HTML and another rendered into an image. Verified on the device: 0 of 10 image words appear anywhere in the tree, 10 of 10 written words do.
Each task runs both ways back to back with the order alternating, so device state cannot load onto one arm. A real phone changes underneath you.
Every task runs on a physical iPhone. Nothing is simulated, and no human decides whether a run passed.
| Model | Pass rate | Passed | Avg steps | Avg time | Cost/task |
|---|---|---|---|---|---|
| Claude Sonnet 5 | 94.8% | 55/58 | 7.2 | 58s | $0.177 |
A task passes only if the phone itself ends in the required state. What the agent says it did counts for nothing: an agent that reports “I have enabled that setting” and an agent that enabled it are different things, and only the second one passes.
A pass rate says how often a model succeeded. This says at what. Each capability is isolated by at least one short task, so a failure names a missing skill instead of pointing vaguely at a long task that went wrong somewhere in the middle.
| Capability | What it takes | Tasks | Passed |
|---|---|---|---|
app-resolve | turning a spoken app name into the right bundle id | 1 | 1/1 |
back | leaving a screen when the way out has no label | 1 | 1/1 |
context-menu | holding a link until its preview menu opens | 1 | 1/1 |
date-navigate | moving around a calendar rather than a list | 1 | 1/1 |
edge-gesture | a swipe that must start at y=0, off the drawn screen | 1 | 0/1 |
hierarchy | going up, not just down, through nested screens | 4 | 4/4 |
icon-only-control | a control with a glyph and no text to match on | 1 | 1/1 |
lifecycle | launching, backgrounding and returning to an app | 2 | 2/2 |
picker | reaching a spinning wheel | 1 | 1/1 |
picker-multi | addressing the right column when several are side by side | 1 | 1/1 |
picker-set | turning a wheel to a value, which a swipe cannot do | 2 | 2/2 |
precise-taps | hitting small targets laid out in a grid | 1 | 1/1 |
read-field | reading a value back off the screen, not guessing it | 2 | 2/2 |
recover | noticing the wrong app is open and fixing it | 1 | 1/1 |
render-dependent | 10 | - | |
scroll-end | reaching something below the fold | 1 | 1/1 |
search-then-act | using an app's own search instead of navigating | 5 | 5/5 |
slider | reaching a continuous control | 1 | 1/1 |
switch | reading and setting a toggle | 1 | 1/1 |
tabbar | moving between tabs in an unfamiliar app | 3 | 3/3 |
text-exact | typing a string that must match character for character | 1 | 1/1 |
text-long | typing something long enough for autocorrect to interfere | 1 | 1/1 |
text-select | selecting text that is already on screen | 1 | 1/1 |
tree-sufficient | 10 | - |
Three parts. The first and the third involve no model at all, which is what makes a score reproducible.
Deterministic steps, no model involved, so every attempt starts identically.
The agent sees an accessibility tree and a screenshot, and acts with taps, typing, gestures and picker wheels.
Assertions read the device directly and decide pass or fail. The agent never sees them.
A complete task, exactly as it is stored in the repository:
id: calculator.arithmetic
name: Do arithmetic in the Calculator
app: com.apple.calculator
difficulty: easy
tags:
- calculator
- input
- first-party
instruction: Open the Calculator app and compute 47 multiplied by 9. Leave the result on screen.
setup:
- terminate: com.apple.calculator
- home
checks:
- kind: foreground_app
text: com.apple.calculator
- kind: regex_on_screen
text: \b423\b
teardown:
- terminate: com.apple.calculator
notes: A precise sequence of taps where every one must land. 47x9=423.
15 easy, 58 medium, 7 hard, all against Apple’s own applications: Settings, Safari, Notes, Reminders, Clock, Calculator, Contacts, Maps, Calendar, Files, Books, Compass, Shortcuts, Voice Memos and the home screen itself. No third-party app is automated, so no third party’s terms are involved.
These are 60 of 76. Sixteen are held back and are not in the public repository or its history: a benchmark whose entire answer key is public becomes training data, and the score then measures memorisation rather than capability. The public 60 cover all 22 capabilities, so a score over them is comparable between models and you can run the whole thing today.
| Task | App | Difficulty | Checks | Steps | Time | Result |
|---|---|---|---|---|---|---|
books.searchOpen the Books app and search for 'Moby Dick'. | iBooks | medium | 2 | 6 | 55s | pass |
calculator.arithmeticOpen the Calculator app and compute 47 multiplied by 9. Leave the result on screen. | calculator | easy | 2 | 9 | 44s | pass |
calculator.chainOpen the Calculator and compute (12 plus 8) multiplied by 5. Leave the answer on screen. | calculator | medium | 1 | 12 | 80s | pass |
calculator.percentageOpen the Calculator and work out what 15 percent of 240 is. Leave the answer on screen. | calculator | medium | 2 | 11 | 63s | pass |
calendar.month_viewOpen the Calendar app and switch it to the month view, showing a whole month at once. | mobilecal | medium | 2 | 3 | 25s | pass |
calendar.todayOpen the Calendar app and make sure it is showing today. | mobilecal | easy | 2 | 3 | 19s | pass |
clock.alarm.reach_pickerOpen the Clock app, go to Alarms, and start adding a new alarm so the time picker is sho | mobiletimer | medium | 2 | 4 | 53s | pass |
clock.alarm.set_timeOpen the Clock app, go to Alarms, start adding a new alarm and set its time to 7:30 in t | mobiletimer | hard | 4 | 6 | 187s | pass |
clock.stopwatch.startOpen the Clock app, go to the Stopwatch tab and start it. | mobiletimer | easy | 2 | 4 | 48s | pass |
clock.tab.alarmsOpen the Clock app and switch to the Alarms tab. | mobiletimer | easy | 2 | 3 | 39s | pass |
clock.tab.timersOpen the Clock app and switch to the Timers tab. | mobiletimer | easy | 2 | 4 | 39s | pass |
clock.timer.setOpen the Clock app and start a timer for 5 minutes. | mobiletimer | medium | 2 | 5 | 53s | fail |
clock.timer.set_hours_minutesOpen the Clock app, go to Timers, and set the timer to 1 hour and 30 minutes. Do not sta | mobiletimer | hard | 3 | 7 | 74s | pass |
clock.timer.set_minutesOpen the Clock app, go to Timers, and set the timer duration to 5 minutes. Do not start | mobiletimer | medium | 2 | 6 | 52s | pass |
clock.worldclock.tabOpen the Clock app and switch to the World Clock tab. | mobiletimer | easy | 2 | 4 | 48s | pass |
compass.read_headingOpen the Compass app and tell me which direction the phone is pointing. | compass | medium | 2 | 3 | 21s | pass |
contacts.searchOpen Contacts and use its search field to search for the letter 'a'. Stop once results s | MobileAddressBook | medium | 2 | 5 | 41s | pass |
files.browse_tabOpen the Files app and go to its Browse tab. | DocumentsApp | easy | 2 | 4 | 30s | pass |
maps.search_placeOpen Maps and search for the Eiffel Tower. Stop once the search results show it. | Maps | medium | 2 | 5 | 47s | pass |
measure.openOpen the Measure app. | measure | easy | 1 | 3 | 25s | pass |
multi.app_switchOpen the Calculator, then open Safari, then go back to the Calculator. | springboard | medium | 1 | 5 | 39s | pass |
multi.return_after_detourOpen the Calculator, then open Notes, then come back to the Calculator. | calculator | hard | 1 | 5 | 31s | pass |
multi.three_appsOpen the Calculator, then the Clock, then Settings. End on Settings. | springboard | medium | 1 | 5 | 45s | pass |
notes.create_titledOpen the Notes app and create a new note whose first line is exactly "phoneshell benchma | mobilenotes | medium | 2 | 7 | 46s | pass |
notes.searchOpen Notes and use search to find the note containing the words 'phoneshell benchmark'. | mobilenotes | medium | 2 | 9 | 56s | pass |
notes.text_selectOpen Notes, create a new note, type the word 'anchor', then long press that word so the | mobilenotes | hard | 2 | 10 | 101s | pass |
notes.type_longOpen Notes, create a new note, and type: the quick brown fox jumps over the lazy dog nin | mobilenotes | medium | 2 | 6 | 45s | pass |
notes.type_punctuationOpen Notes, create a new note, and type exactly: benchmark-test (v2). | mobilenotes | medium | 2 | 6 | 36s | pass |
probe.image.eightOpen https://blolabel.ai/probe/eight/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.fiveOpen https://blolabel.ai/probe/five/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.fourOpen https://blolabel.ai/probe/four/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.nineOpen https://blolabel.ai/probe/nine/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.oneOpen https://blolabel.ai/probe/one/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.sevenOpen https://blolabel.ai/probe/seven/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.sixOpen https://blolabel.ai/probe/six/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.tenOpen https://blolabel.ai/probe/ten/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.threeOpen https://blolabel.ai/probe/three/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.image.twoOpen https://blolabel.ai/probe/two/ and tell me the word shown in the picture. | mobilesafari | medium | 2 | - | - | not run |
probe.text.eightOpen https://blolabel.ai/probe/eight/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.fiveOpen https://blolabel.ai/probe/five/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.fourOpen https://blolabel.ai/probe/four/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.nineOpen https://blolabel.ai/probe/nine/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.oneOpen https://blolabel.ai/probe/one/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.sevenOpen https://blolabel.ai/probe/seven/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.sixOpen https://blolabel.ai/probe/six/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.tenOpen https://blolabel.ai/probe/ten/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.threeOpen https://blolabel.ai/probe/three/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
probe.text.twoOpen https://blolabel.ai/probe/two/ and tell me the written word. | mobilesafari | medium | 2 | - | - | not run |
reminders.createOpen Reminders and add a new reminder called 'collect parcel'. | reminders | medium | 2 | 6 | 52s | fail |
resilience.deep_startOpen Safari and go to example.com | mobilesafari | medium | 1 | 3 | 29s | pass |
resilience.scrolled_listIn Settings, open the General page. | Preferences | medium | 2 | 5 | 43s | pass |
resilience.wrong_appGo to the Settings page for Display and Brightness. | Preferences | medium | 2 | 13 | 69s | pass |
safari.link_menuOpen Safari, go to example.com, and long press the 'More information...' link until its | mobilesafari | hard | 2 | 4 | 32s | pass |
safari.navigate.exampleOpen Safari and go to example.com | mobilesafari | easy | 2 | 3 | 40s | pass |
safari.tabs.open_switcherOpen Safari and show the tab switcher, where all the open tabs are listed. | mobilesafari | medium | 2 | 4 | 27s | pass |
safari.read.headingOpen Safari, go to example.com, and tell me the exact heading shown on that page. | mobilesafari | medium | 1 | 3 | 22s | pass |
safari.search.webOpen Safari and search the web for 'iso 8601 date format'. Stop once results are showing | mobilesafari | medium | 2 | 5 | 37s | pass |
settings.about.reachOpen Settings and navigate to the About page that shows the iPhone's model number. | Preferences | easy | 2 | 6 | 38s | pass |
settings.about.serialOpen Settings and find the About page, then tell me the Model Number of this iPhone. | Preferences | medium | 2 | 6 | 40s | pass |
settings.accessibility.reachOpen Settings and go to the Accessibility page. | Preferences | easy | 2 | 23 | 109s | pass |
settings.back.twiceOpen Settings, go into General, then into About, then go back to the main Settings list. | Preferences | medium | 3 | 9 | 78s | pass |
settings.battery.percentageOpen Settings, go to the Battery screen, and tell me the current battery percentage. | Preferences | medium | 2 | 11 | 55s | pass |
settings.bluetooth.reachOpen Settings and go to the Bluetooth page. | Preferences | easy | 2 | 4 | 26s | pass |
settings.datetime.reachOpen Settings and navigate to the Date & Time page under General. | Preferences | medium | 2 | 10 | 68s | pass |
settings.fonts.reachOpen Settings and navigate to the Fonts page under General. | Preferences | medium | 2 | 7 | 61s | pass |
settings.keyboard.reachOpen Settings and navigate to the Keyboard page under General. | Preferences | medium | 2 | 8 | 71s | pass |
settings.scroll.to_bottomOpen Settings and scroll all the way to the very bottom of the main settings list. | Preferences | medium | 2 | 7 | 69s | pass |
settings.search.wifiOpen Settings and use its search field to search for "Wi-Fi", then open the Wi-Fi settin | Preferences | medium | 2 | 6 | 45s | pass |
settings.search.batteryOpen Settings, search for "Battery" using the search field, and open the Battery page fr | Preferences | medium | 2 | 6 | 49s | pass |
settings.search.bluetoothOpen Settings, search for "Bluetooth" using the search field, and open the Bluetooth pag | Preferences | medium | 2 | 26 | 199s | pass |
settings.search.storageOpen Settings, search for "Storage" using the search field, and open the Storage page fr | Preferences | medium | 2 | 5 | 31s | pass |
settings.software_update.reachOpen Settings and navigate to the Software Update screen. Do not install anything. | Preferences | medium | 2 | 8 | 49s | pass |
settings.storage.reachOpen Settings and navigate to the iPhone Storage screen that shows how much space is use | Preferences | medium | 2 | 7 | 38s | pass |
settings.textsize.reachIn Settings, reach the screen with the slider that makes text larger. Do not change it. | Preferences | hard | 2 | 23 | 179s | pass |
settings.wifi.switch_visibleOpen Settings and go to the Wi-Fi page. | Preferences | easy | 2 | 4 | 29s | pass |
system.appstore.openOpen the App Store app. Do not download anything. | AppStore | easy | 1 | 3 | 22s | pass |
system.notifications.openOpen the iPhone's Notification Centre. | springboard | hard | 1 | 24 | 232s | fail |
system.spotlight.find_appUsing the iPhone's own search (Spotlight), find the Calculator app and open it. | springboard | medium | 1 | 7 | 54s | pass |
weather.openOpen the Weather app. | weather | easy | 1 | - | - | not scored |
weather.read_temperatureOpen the Weather app and tell me the current temperature where I am. | weather | medium | 2 | - | - | not scored |
A Mac, an iPhone, a cable and an Apple developer account. The harness builds and installs the automation runner onto the phone for you.
git clone https://github.com/Blomega/phoneshell
cd phoneshell
bin/phoneshell doctor # tells you exactly what is missing
bin/phoneshell setup # builds and installs the runner on your iPhone
bin/phoneshell up # brings the bridge up
bin/phoneshell bench --list
bin/phoneshell bench --model claude-sonnet-5The harness is free and open. What breaks it is not your code: it is an iOS point release, an Xcode update or a change in WebDriverAgent, any of which can stop the bridge working on a Tuesday morning. We keep a tested build working and there is a person to call.
Cloud farms give you clean, ephemeral phones. Some things need a real handset: an OTP to a real SIM, a payment through a local wallet, region-locked content, an account with history behind it.
Every published mobile-agent number is Android, because Android runs in software and iOS does not. If you want your model measured on iOS rather than assumed to transfer, the suite above is free to run today.