Agents that operate computers are measured on desktops and on Android, because both can be run in software, thousands at a time, for free. An iPhone cannot. It needs a physical device, a Mac, a signed developer build and a harness that does not lie to you. So nobody measures it. This is the missing benchmark.
Every task is run on a physical iPhone running iOS 26. Nothing is simulated, and no human decides whether a run passed.
| Model | Pass rate | Passed | Avg steps | Avg time | Cost/task |
|---|---|---|---|---|---|
| Claude Sonnet 5 | 100.0% | 3/3 | 8.3 | 44s | $0.169 |
Each task is three parts. The first and third involve no model at all, which is what makes a score reproducible.
Deterministic steps put the phone in a known state, so every attempt starts identically.
One sentence is handed to the agent. It sees the phone through an accessibility tree and a screenshot, and acts with taps, typing and gestures.
Assertions read the phone directly and decide pass or fail. The agent never sees them.
A complete task, exactly as it is stored in the repository:
id: calculator.arithmetic
name: Do arithmetic in the Calculator
app: com.apple.calculator
difficulty: easy
tags:
- calculator
- input
- first-party
instruction: Open the Calculator app and compute 47 multiplied by 9. Leave the result on screen.
setup:
- terminate: com.apple.calculator
- home
checks:
- kind: foreground_app
text: com.apple.calculator
- kind: regex_on_screen
text: \b423\b
teardown:
- terminate: com.apple.calculator
notes: A precise sequence of taps where every one must land. 47x9=423.
9 easy, 18 medium, 5 hard. All of them run against Apple's own applications: Settings, Safari, Notes, Reminders, Clock, Calculator, Weather, Contacts and the home screen itself. No third-party app is automated, so no third party's terms are involved.
| Task | App | Difficulty | Checks | Result |
|---|---|---|---|---|
calculator.arithmeticOpen the Calculator app and compute 47 multiplied by 9. Leave the result on screen. | calculator | easy | 2 | pass |
calculator.percentageOpen the Calculator and work out what 15 percent of 240 is. Leave the answer on screen. | calculator | medium | 2 | pass |
clock.stopwatch.startOpen the Clock app, go to the Stopwatch tab and start it. | mobiletimer | easy | 2 | pass |
clock.timer.setOpen the Clock app and start a timer for 5 minutes. | mobiletimer | medium | 2 | not run |
clock.worldclock.tabOpen the Clock app and switch to the World Clock tab. | mobiletimer | easy | 2 | not run |
contacts.searchOpen Contacts and use its search field to search for the letter 'a'. Stop once results s | MobileAddressBook | medium | 2 | not run |
multi.app_switchOpen the Calculator, then open Safari, then go back to the Calculator. | springboard | medium | 1 | not run |
multi.settings_to_safariFind this iPhone's iOS version in Settings, then open Safari and search the web for that | springboard | hard | 2 | not run |
notes.checklistOpen Notes, create a new note, and make its first line a checklist item reading 'milk'. | mobilenotes | hard | 2 | not run |
notes.create_titledOpen the Notes app and create a new note whose first line is exactly "phoneshell benchma | mobilenotes | medium | 2 | not run |
notes.searchOpen Notes and use search to find the note containing the words 'phoneshell benchmark'. | mobilenotes | medium | 2 | not run |
reminders.completeOpen Reminders, find the reminder called 'collect parcel' and mark it as completed. | reminders | hard | 2 | not run |
reminders.createOpen Reminders and add a new reminder called 'collect parcel'. | reminders | medium | 2 | not run |
resilience.scrolled_listIn Settings, open the General page. | Preferences | medium | 2 | not run |
resilience.wrong_appGo to the Settings page for Display and Brightness. | Preferences | medium | 2 | not run |
safari.navigate.exampleOpen Safari and go to example.com | mobilesafari | easy | 2 | not run |
safari.newtabOpen Safari and open a brand new empty tab. | mobilesafari | medium | 2 | not run |
safari.read.headingOpen Safari, go to example.com, and tell me the exact heading shown on that page. | mobilesafari | medium | 1 | not run |
safari.search.webOpen Safari and search the web for 'iso 8601 date format'. Stop once results are showing | mobilesafari | medium | 2 | not run |
settings.about.reachOpen Settings and navigate to the About page that shows the iPhone's model number. | Preferences | easy | 2 | not run |
settings.accessibility.reachOpen Settings and go to the Accessibility page. | Preferences | easy | 2 | not run |
settings.battery.percentageOpen Settings, go to the Battery screen, and tell me the current battery percentage. | Preferences | medium | 2 | not run |
settings.bluetooth.reachOpen Settings and go to the Bluetooth page. | Preferences | easy | 2 | not run |
settings.search.wifiOpen Settings and use its search field to search for "Wi-Fi", then open the Wi-Fi settin | Preferences | medium | 2 | not run |
settings.software_update.reachOpen Settings and navigate to the Software Update screen. Do not install anything. | Preferences | medium | 2 | not run |
settings.storage.reachOpen Settings and navigate to the iPhone Storage screen that shows how much space is use | Preferences | medium | 2 | not run |
system.app_switcher.openOpen the iPhone's app switcher so the recently used apps are showing. | springboard | hard | 1 | not run |
system.appstore.openOpen the App Store app. Do not download anything. | AppStore | easy | 1 | not run |
system.control_centre.openOpen the iPhone's Control Centre. | springboard | hard | 1 | not run |
system.spotlight.find_appUsing the iPhone's own search (Spotlight), find the Calculator app and open it. | springboard | medium | 1 | not run |
weather.openOpen the Weather app. | weather | easy | 1 | not run |
weather.read_temperatureOpen the Weather app and tell me the current temperature where I am. | weather | medium | 2 | not run |
You need a Mac, an iPhone, a cable and an Apple developer account. The harness builds and installs the automation runner onto the phone for you.
git clone https://github.com/<org>/phoneshell
cd phoneshell
bin/phoneshell doctor # tells you exactly what is missing
bin/phoneshell setup # builds and installs the runner on your iPhone
bin/phoneshell up # brings the bridge up
bin/phoneshell bench --list
bin/phoneshell bench --model claude-sonnet-5The harness is free and open, forever. What breaks it is not your code: it is an iOS point release, an Xcode update, or a change in WebDriverAgent, any of which can stop the bridge working on a Tuesday morning. We keep a tested build working and there is a person to call. That is what is sold, and nothing else is.
Cloud device farms give you clean, ephemeral phones. Some things cannot be tested on one: an OTP to a real SIM, a payment through a local wallet, region-locked content, an account with real history behind it. Those run on a real handset or they do not run. If that is your release checklist, this is built for exactly that.
Every published mobile-agent number is Android, because Android runs in software and iOS does not. If you want your model measured on iOS instead of assumed to transfer, the suite above is free to run today, and held-out tasks and scored runs on real hardware can be arranged.