What we measured on real hardware, what broke, and what the numbers actually support. Every claim here is traceable to a run or a source.
Success Is Not Self-Explanatory (arXiv:2607.24054, 27 July 2026) measures a 19.1 to 25.9 percentage point gap between a correct and a matched incorrect value exposed to an agent under otherwise identical conditions. phoneshell's only check that grades a model's words rather than the phone's state sidesteps that exact confound by fixing ground truth on a page it hosts itself.
WWDC26 session 343 shows apps declaring a persistent EntityIdentifier so Siri can resolve on-screen references without reading pixels or a tree. phoneshell drives ordinary apps that never opt in, and its memory.py quotes a real WebDriverAgent identifier baking in a Bluetooth MAC address and a raw pointer that has to be stripped before two visits to a screen can be treated as the same screen.
LearnAct (arXiv:2504.13805) retrieves demonstration knowledge by embedding a new task instruction and ranking by cosine similarity, lifting Gemini-1.5-Pro from 19.3% to 51.7% offline on tasks never attempted before. phoneshell memory.py only activates once a screen fingerprint repeats, contributing nothing on a task family's first appearance.
A3 (arXiv:2501.01149) explicitly rejects device-state checks for its 100-task benchmark, judging progress from screenshots alone via an MLLM reward model (53% SR / 66.4% ESAR, best agent). phoneshell makes the opposite architectural bet for the same class of problem, preferring device state and falling back to pixels only when state is unavailable.
AndroidControl-Curated (arXiv:2510.18488) sorts AndroidControl's label errors into three categories covering 24.13%, 8.12% and 37.63% of reviewed samples, lifting curated success rates 11 to 18 points for five models. phoneshell's device-state checker has no recorded action field to be wrong about in the first place.
DeltaReplay (arXiv:2610.11707) gains up to 10.3 points on AndroidWorld and 25.0 on SPA-Bench by deciding per step whether a stored route matches a new task closely enough to replay, replay with swapped parameters, or hand off to the base agent, with measured 75 to 84 percent precision on that decision. phoneshell's get(name) macro lookup has no equivalent: a macro runs only when called by its exact saved name.
A July 2026 paper (arXiv:2608.24898) proposes a Model Context Protocol layer for accessibility trees, scoring it Low integration cost against High for today's per-framework scraping in its own comparison table, with no reference implementation built. phoneshell's perception/tree.py is the per-framework side of that table made concrete: rules written specifically for how Apple's XCUITest reports iOS UI.
AndroidReality (arXiv:2608.07775) measures 13 to 36 point success-rate drops across eight Android agents when notifications, permission requests and security sheets interrupt a run, with UI-TARS-1.5-7B losing 36.61 points specifically to that perturbation class. phoneshell's own overlay-dismissal reflex deliberately excludes permission dialogs from auto-handling by design, and its separate tap-confirmation gate has no pattern that would catch the model doing it instead.
GMA (arXiv:2608.27477) finds all 8 tested frontier models collapse on its Realistic task tier, and its own harness ablation shows adding text captions to a fixed 2 screenshots nearly tripled the Realistic score (11.0 to 34.6), beating an 8x increase in image count. phoneshell's observation.py already sends a text tree serialization and a screenshot on every step but has no code path that captions the image.
MyPCBench's six agents average 43.7-69.2 steps per task, and even zero-score failures average 22-85 steps before quitting. phoneshell's own KeepAwake class, built after this project lost four benchmark runs to the iPhone's screen locking mid-task, is called from exactly one feature (app crawling) and never from bench/runner.py, the code path that runs phoneshell's 80 scored tasks.
RiOSWorld measures a 91.3% intention and 84.0% completion unsafe rate for induced OS-operation tasks, and 41.1%/33.3% for file writes, across ten multimodal computer-use agents. phoneshell's own wda/patches.py documents two custom WebDriverAgent routes, including one that writes to the camera roll, both withoutSession and reachable by anything on the network when the phone is driven over wifi, because WebDriverAgent authenticates nothing.
UI-CUBE (UiPath) scores five computer-use agents at 66.7-84.8% on simple UI tasks and 9.5-19.4% on complex workflows built around copy-paste between apps, against a 97.9%/61.2% human baseline. phoneshell/gestures.py already implements the iOS three-finger pinch gesture for copy and paste, but no task in its 80-task public suite ever calls it.
GhostEI-Bench measures a Vulnerability Rate from 16.43% (GPT-5) to 56.25% (UI-TARS-7B-SFT) against injected overlay attacks on Android GUI agents. phoneshell's own overlay-dismissal reflex refuses by construction to tap the same attack words, falls back to three content-blind physical gestures, and only then hands an unresolved overlay to the model, which is the exact decision point GhostEI-Bench's numbers say models get wrong.
Computer Use at the Edge of the Statistical Precipice (arXiv:2605.08261, Meta) shows a no-vision replay script beating a frontier model on two static benchmarks and collapsing on the reconfigurable DigiWorld. phoneshell's deterministic setup step guarantees the exact property that makes the exploit possible, and has never been tested for it, even though phoneshell's own stats module already implements the paper's other prescription (Wilson intervals, paired exact tests).
MyPhoneBench (arXiv:2604.00986) scores five models on 300 tasks twice: task success and data-handling privacy. Claude Opus 4.6 ranks first on success and fourth on privacy; the combined ranking reshuffles. phoneshell's nine check kinds are entirely task-specific and opt-in, with no second, independent axis.
D-GARA (arXiv:2511.16590, AAAI 2026) interrupts five GUI agents mid-task with ordinary anomalies (permission dialogs, battery warnings, crashes); the worst model loses 32.9 points. phoneshell already detects the same kind of system alert on a real iPhone but its results schema has no field recording whether one occurred.
EntWorld (arXiv:2601.17722) ran a screenshot-plus-tree ablation on three models; combining inputs gained 18.5 points for a model fine-tuned on the benchmark's own apps but cost 4.9 points for the untuned model in the same table. phoneshell's observation code quotes only the first number, uncredited.
A Permission Literacy study (arXiv:2608.04755, 5 August 2026) injects permission pop-ups into real GUI tasks across four frontier multimodal LLMs and finds an App-Trust Bias: holding a Calendar task fixed and changing only the requesting app from Calendar to PiMusic drops grants from 26 of 32 to 0 of 32. phoneshell's 14-pattern confirm gate checks only the tapped label's text, with no requester field and no match for a permission alert's own buttons.
V-Droid (arXiv:2503.15937, accepted to MobiCom 2026) replaces direct action generation with an LLM verifier that scores discretized candidate actions: 59.5% on AndroidWorld (+5.2 points), 38.3% on AndroidLab (+2.1), 49% on MobileAgentBench (+9), at 4.3 seconds a step. phoneshell's MCP server hands the screen to whichever model drives it and executes whatever action comes back, with no verifier in between.
PrefDiag-Bench (arXiv:2610.30573, 22 October 2026) runs 1,664 stage-wise probes across 5 preference types and 11 agent systems; even Seed-2.0-Pro, its best model, clears only 47.17% of Process preferences with an explicit profile supplied, and handing agents relevant evidence directly raises end-to-end success by up to 7.06 points. phoneshell's 80 tasks carry no persona, behavioral log or preference field of any kind.
Xiaomi-GUI-0 (arXiv:2606.31410) trains and evaluates closed-loop on physical phones, then scores 72.0% on its own RealMobile benchmark against 78.9% on the public AndroidWorld, a 6.9-point gap in the harder benchmark's disfavor. phoneshell's six-model, 58-task board on first-party iOS apps sits at 89.7% to 96.6% with zero of fifteen pairs separating, the same ceiling effect from the opposite direction.
PhoneHarness (arXiv:2606.14832) reports a 75.0% headline pass rate, 12.9 points ahead of the best non-PhoneHarness baseline, while its own model-combination table lists 74.8% for the same configuration. phoneshell's five asserted check kinds have no analogue to PhoneHarness's tool-called or email-recipient checks, because phoneshell's action module has no Tool action type at all.
AndroidDaily (arXiv:2605.27761) scores agents on 350 closed-source Android apps using GRADE, a tiered guideline judge reporting 87.37% human agreement, arithmetically a 12.63% (about 44-task) disagreement rate. phoneshell's own 153 assertions over 80 tasks ask a model to judge anything in only 20 of them; the rest read device state directly.
Beyond the GUI Paradigm (arXiv:2606.19388) shows Claude Code with ADB shell access beating every reproducible GUI baseline on AndroidWorld and MobileWorld, with an oracle CLI ceiling of 88.8% / 86.3% at 10.7 steps per task versus 18.6 for GUI. Every figure is Android-only; phoneshell's action module has no verb that issues a shell command, by platform constraint rather than oversight.
iOSWorld (arXiv:2606.09764) reports its best configuration at 52% overall / 37% multi-app in its arXiv abstract and at 51% / 36% in its own ICML 2026 workshop listing for the same run. phoneshell's board uses a different task set, app population and device, so the point here is only that iOSWorld's own two citations disagree with each other.
MVISU-Bench's Section 5.1.4 defines Unethical-category success as completing the harmful step, while Section 4.2 calls its Aider module a Risk Mitigator meant to prevent exactly that; Table 4 credits Aider with raising that score from 65.71% to 97.14% (EN) and 22.22% to 97.22% (CN). phoneshell's Guard never reads the instruction text at all, only tap labels and app bundle IDs.
ScaleWoB measures a 92.08% human success rate on its own 120-task mobile benchmark against a 27.92% average for five SOTA agents, falling to 17.82% on long-horizon tasks. phoneshell's results.json has no equivalent human-baseline row for its own 80 tasks.
MobiFlow's introduction/Table 3 give UI-TARS-1.5 a 60.4% success rate, but Section 5.2 calls 60.4 GUI-Owl's completion rate, a figure Table 3 itself puts at 77.9 for GUI-Owl. phoneshell avoids the whole problem MobiFlow's trajectory-fusion graph solves by reading a real device's OS state directly.
MobilePA-Bench scores mobile planner agents on four weighted capabilities (tool use 50%, memory 20%, skill 20%, sub-agent collaboration 10%); GPT-6-Astra's 76.87% is that composite, and 9 of 14 models score below 70%. phoneshell's schema has one pass/fail result per task and no capability breakdown.
iOSWorld (arXiv:2606.09764, 8 June 2026) splits 133 tasks into single-app (81.5%), multi-app (36.7%) and memory/personalization (54.3%) for its best configuration; accessibility-tree access adds up to 26 points for frontier models and nothing for smaller ones. phoneshell has no persona, no multi-app task, and no cross-app memory category in its schema.
WWDC26 session 295 introduces AppIntentsTesting, which executes App Intents across process boundaries and asserts on typed results with no UI automation anywhere in the test. phoneshell's closest equivalent, a shortcuts://run-shortcut deeplink, is an untyped URL string that returns nothing phoneshell can check.
Appium 4 beta 4 (9 Oct 2026) lets the XCUITest driver control a real iOS 18+ device over a tunnel from any host, with no Xcode in the driving loop, provided WebDriverAgent is already installed. phoneshell's own six-step bootstrap needs Xcode at two of those steps, including the xcodebuild call that builds WDA in the first place.
MobileJailBench (arXiv:2610.23303, 16 Oct 2026) uses Failure-Distilled Jailbreak Evolution to raise every one of ten tested models' jailbreak attack success rate, Gemini-2.5-Pro the most, 50.7% to 83.6%. phoneshell's own Guard class has exactly one hard block, a 3-bundle-ID credential denylist, and has never been adversarially tested.
Computer Agent Arena, an ICLR 2026 platform, collects 2,201 human votes over 12 computer-use agents and reports ranking reversals against static benchmarks. phoneshell's own static leaderboard failed differently: a McNemar analysis found three of six models statistically tied, with no size of sweep able to separate them, a validity failure and a power failure that neither project has tested against the other's data.
WeaveBench pairs 114 hybrid GUI-and-CLI tasks with a trajectory-aware judge designed to catch fabricated screenshots and hardcoded metrics, reporting a 41.2% best pass rate. phoneshell's device-state checks sidestep the problem by never reading the agent's own report, a structurally different defense that has never been tested against a model deliberately trying to fake a pass.
MobileBench-OL's noise-injection subset injects one of four named interruption types (repeat, unexecuted, delay, pop-up) at a 20% chance per step, dropping its best agent from 60.97% to 56.77%. phoneshell has never injected noise deliberately, yet its own day-apart rerun of 144 identical tasks flipped 12 outcomes with no equivalent breakdown by cause.
OSWorld 2.0's 108 long-horizon desktop tasks average 318 tool calls with Claude Opus 4.7 and take a human a median 1.6 hours, scored to a 500-step ceiling where the best agent still completes only 20.6%. phoneshell's own hard cap stops at 60 steps, a value no phoneshell task has ever come close to needing.
MobiBench scores GPT-4.1's Android task success at 27.17% reading the accessibility tree as text and only 4.72% reading a raw screenshot alone, a result about which input helps an agent choose its next tap. phoneshell's own vision finding, 100% versus 0% on an iOS screen, answers a different question entirely: whether the thing checking the outcome needs to see the pixels at all.
VenusBench-Mobile's best agent pairing scores 36.9% on single-run Android tasks but only 15% when a task must succeed on all five of its variants. phoneshell's own day-apart rerun of 144 identical iPhone tasks flipped 12 outcomes, 11 in the same direction, traced to the harness rather than the model. Two different kinds of instability, neither visible in a single success rate.
A11y-Compressor's learned pipeline restructures desktop accessibility trees to 22% of their tokens while gaining 5.1 points of task success on OSWorld. phoneshell's hand-pruned iOS tree reads cut wall time and token cost by a similar order of magnitude but never measured whether the one element they stopped describing ever mattered to a task.
A CHI '26 dataset shows computer-use agent success declining gradually as access narrows (78.3% to 41.6% to 28.3%). phoneshell's own pre-registered iOS probe finds no gradient at all: a screenshot is worth 100% or 0% per screen, because it isolates one channel on tasks engineered so the answer is entirely inside or outside it.
DAC-Bench (ACL 2026) scores seven agents across 830 episodes and four compositional task structures: success falls below 5% on 6 to 8 step tasks and branch accuracy averages 38%. phoneshell's own suite has exactly three tasks built from the Sequential structure that taxonomy names, and its checkers grade them with one assertion that cannot see a branch at all.
esp32-iphone-agent (MIT, about 3,300 GitHub stars) controls a real iPhone through AirPlay mirroring and a $7 ESP32-S3 acting as a USB keyboard and mouse, with no Mac-side Xcode, developer account or accessibility-tree read described in either write-up we checked. phoneshell reads the tree on every step and grades its 80 tasks from device state because, by its own runner's rule, the agent's account of what it did counts for nothing.
pymobiledevice3 11.40.1 (20 October 2026, PR #2069) makes a stalled RemoteServiceDiscoveryService connection fail after 10 seconds instead of waiting forever. phoneshell shells out to that exact connection through developer dvt xcuitest, and its own 85-second give-up path in cli.py never terminates the subprocess it spawned.
WebDriverAgent's MJPEG broadcaster bound to every network interface regardless of USE_IP until pull request #1299 landed on 16 October 2026. phoneshell sets USE_IP and forwards both WDA ports, but its lan_exposure_check has only ever opened a socket to port 8100, the port that was never the problem.
MobileGym trains Qwen3-VL-4B-Instruct with GRPO inside a parallel Android simulator, gaining 42.8 points on a 59-task subset, then retains 40.7 of those points (95.1%) when the same tasks run on a physical Redmi Note 12 Turbo. phoneshell skips simulation entirely and runs all 80 of its tasks on a physical iPhone 17 Pro Max from the start, trading a simulator's parallelism for having no sim-to-real gap to measure.
A measurement-theory audit of agent benchmarks finds ComtradeBench's production harness, not the model, decides retry, dedup, totals-drop, pagination, and submission, producing byte-identical outputs across 21 of 21 tested model pairs, and a well-formed fabrication scoring 0.987 under the shipped judge, tied with the correct answer. phoneshell's published scaffold prompt covers tool-batching etiquette only and makes none of a task's decisions.
DailyDroid finds a 5.3-point (GPT-4o) and 4.0-point (o4-mini) average success gain from adding a screenshot to 75 everyday Android tasks, but the largest failure category, UI Retrieval, has the identical failure count in both conditions. phoneshell's own screenshot result is binary (0% or 100%) because it isolates pixel-only tasks instead of averaging across a mixed suite.
A3 grades the same 100 live-app tasks with a strict pass/fail rate and a more forgiving MLLM-judged essential-state score, validated at 95.7% precision against human labels; nearly every agent scores roughly double under the judge. phoneshell's five check kinds read the accessibility tree and screen text directly, with no judge model anywhere in the loop.
GMA's harness ablation shows structured state tracking and multi-image context retention each move Qwen3.7-Plus's success rate on their own, separate from the model. phoneshell's published scaffold prompt only tells the model to batch tool calls and skip redundant observation; it does neither of GMA's two levers.
ProBench splits mobile tasks into state-related and process-related categories; its best model, Gemini 2.5 Pro, drops 17.7 points between them. phoneshell's five check kinds all run once, after the agent finishes, and never inspect an intermediate screen.
MAS-Bench pairs 139 mobile tasks with 88 predefined shortcuts (11 APIs, 70 deep links, 7 RPA scripts) and finds hybrid agents beat GUI-only agents by up to 44.7 points on cross-app tasks. phoneshell's action space has exactly one shortcut, a deep-link opener, and no catalog to draw from.
MemGUI-Bench names five memory-hallucination failure modes that account for 58.9% of 11 agents' non-timeout failures, and finds Agent-S2's pass@3 falls 20 points without long-term memory. phoneshell persists screen memory across runs but always re-resolves a remembered control against the live accessibility tree before acting on it.
UI-Venus-2's technical report scores models on MobileWorld at both a 50-step and a 100-step budget; UI-Venus-2-9B gains 9.4 points, UI-Venus-2-27B gains 6.8, Qwen-UI-Agent-27B gains 3.4. All 80 of phoneshell's tasks run at the schema default of 25 steps and have never been re-run at a second budget.
VenusBench-Mobile (ICML 2026) retests each of its 149 base tasks in Chinese, in a paraphrase, in dark theme, and on a tablet; the best agent's Stability Pass Rate across all four is 15%, most score 0%. None of phoneshell's 80 environment files vary theme, language, or device shape.
Qwen-UI-Agent's technical report adds cli_command, direct bash execution, to the same action space as its GUI taps, trained across more than 100 physical devices and up to 10,000 parallel sandboxes on trajectories exceeding 100 turns. Every action phoneshell's actions.py exposes to an agent is a WebDriverAgent touch, type or swipe call; none of them opens a shell.
Posted 13 October 2026, GroundGUI pairs a 586K-screenshot dataset with a distillation-to-reinforcement-learning method to train an 8B grounding model, scoring 38.3% to 95.2% across five benchmarks. phoneshell reads the iOS accessibility tree directly and falls back to a zero-training template correlator only when the tree is empty.
An ACL 2026 benchmark of 830 episodes across 35 Android and iOS apps finds seven agents' success collapsing below 5% on 6-8 step branching tasks, with branch accuracy averaging 38%. phoneshell's 80 tasks, read from their schema, cannot express a branch at all: every one is a single straight line from setup to checks.
A method published 16 September 2026 restores a long-horizon agent to an earlier checkpoint and keeps what the failed attempt learned, gaining 3.09 to 6.57 points over the strongest baseline on three benchmarks. phoneshell's own step-budget code treats running out of turns as an outright failure, with no saved state and nothing to roll back to.
BlueLM-GUI's technical report describes a fleet of hundreds of real phones feeding a continual pre-training, fine-tuning and reinforcement learning pipeline, with roughly 5,000 targeted anomalous-state samples across 14 classes folded back into training. phoneshell runs on one real iPhone and keeps only a pass or fail row per attempt; nothing it records ever retrains anything.
pymobiledevice3 fixed a macOS 27 kernel regression that broke its default developer-service tunnel in v11.13.0. phoneshell's requirements.txt pins a version nine days older than that fix and 59 releases behind, across two bring-up checks that fall in the affected command family; this is a version comparison, not a reproduced failure.
AmbiBench finds Task Success Rate falls to 0% for two tested agents on its most ambiguous mobile instruction tier, with clarification rates as low as 2.4%. All 80 of phoneshell's tasks sit at the benchmark's most explicit tier, and its only confirmation gate checks for risk, never for comprehension.
An audit of 150 FAIL verdicts across five computer-use benchmarks found 15.3% wrong, 10.7% of them evaluator false negatives. phoneshell's own task files document three such false negatives already fixed, and two of its three current failures share their shape with no retained evidence to settle which.
WDA 16.15.0 added an opt-in flag so a later app launch can claim WDA's tracked "application under test" status, gaining quiescence inheritance, alert monitoring and crash detection. phoneshell already duplicates the first two itself; it has nothing for crash detection, and never sets the flag.
Two agents, two different answers to the same fact: an iPhone has one screen and one owner. One watches for that owner continuously. The other only knows another of its own processes exists. iOS 26 has one interactive display and one digitizer, so a phone's owner and an agent driving it cannot act at once. phoneshell's Coexistence watchdog samples the foreground app and a screenshot every 0.8 seconds and waits 6.0 seconds of continued quiet before resuming. iPhone Use, created 6 October 2026, holds a single fcntl lock that errors on a second concurrent call and has no concept of a human owner.
WebDriverAgent 16.15.0 (12 October 2026, PR appium/WebDriverAgent#1289) adds an asAppUnderTest flag because an app launched without it gets no quiescence wait, crash detection or alert redirect; the PR's own test found 3 of 3 post-launch lookups failing without the flag and 0 of 3 failing with it. phoneshell's open_app() already tries launchUnattached and activate before ever calling the endpoint the fix touches, and a code comment names the exact symptom this was built to dodge.
AgentHorizon (arXiv:2610.11050, 8 October 2026) pairs each recording with a related instruction it does not satisfy; the best judge, GPT-5.5 in Codex, reaches 80.9% balanced accuracy. Run the same swap over phoneshell's 80 task files: 46 task pairs are implied by check text alone, 40 of them wrong answers, across 11 tasks that passed 66 of 66 runs on 9 September.
nanoMuse (arXiv:2610.08699, 6 October 2026, v0.1.40) runs one open personal agent on every device but leaves its iOS app without hands, because iOS lets no app operate another. phoneshell drives an iPhone from a Mac through an XCUITest runner and scores 322 of 349 device-graded runs. nanoMuse's Sentinel gates a Send tap and a typing step that submits; phoneshell's gate stops neither.
A merge rule that ignores how screens look and compares what their actions do lifted a frozen Qwen3-8B from 72.3% to 78.2% on ten MiniWoB++ tasks (arXiv:2609.38778), where merging by appearance scored 70.8%. On phoneshell's eleven crawls of 11 September 2026, its appearance-only fingerprint gave 29 titled screens 60 keys, 14 for Settings alone, while only 4 of 38 repeated taps behaved differently, all on promotional rows.
DIBench (arXiv:2610.06898) shows deceptive UI content lifting a mobile agent's completion from 42.0% to 72.4% while steering 45.8% of its choices; PAIR (arXiv:2610.07972, 6 October 2026) finds subgoal success 7.0 to 15.4 points lower where the screen depends on the user. No phoneshell task asks an agent to commit to one item, and 12 checks pass on the opened app or the typed query. On the 9 September run those 12 tasks gave 71 of 322 passes.
WebDriverAgent 16.14.1 (7 October 2026) exempts com.apple.WebKit processes from the accessibilityDeadline idle check, which crashed WKWebView content and left pages blank (issue #1290). phoneshell has run with the setting at 10 s on 16.12.4 since 8 September. In 189 published web-content runs its grader found the page heading in the tree 29 of 30 times, and the text probes passed 30 of 30 with no screenshot.
Gan et al. (arXiv:2610.01491) re-reviewed every WebArena-Lite failure for GPT-5.5 and recovered 9 to 14 successes per setting that the evaluator rejected. phoneshell's two public cross-model rounds have 52 failures; 33 end with the model claiming success, 16 of them on two tasks where only the outcome check failed. No screenshot or screen tree was kept, so none can be audited.
Control OSWorld (arXiv:2610.03818, 2 October 2026) pairs 318 OSWorld tasks with 81 hidden harmful side tasks. A Claude Haiku 4.5 monitor reading the whole trajectory catches 85% at a 1% false-positive rate; scoring each step before it runs, without the agent's visible text, it catches 8%. That second setup is the only one a phone agent's MCP server can build, and phoneshell's audit log keeps less than it.
Infinite-Dreamer (arXiv:2610.05861) and AutoGUIWorld (arXiv:2610.01215) train GUI agents on screenshots an image model generated. Synthetic-only data lifts Qwen3-VL-8B on AndroidWorld from 39.37 to 43.82, but only 9.3% of imagined rollouts pass the filter for ten straight steps, and hard tasks get worse. Both judge transitions with a VLM; phoneshell's real iPhone crawls hold 237 device-judged tap transitions, 41 of which changed nothing.
DeskForge (arXiv:2610.02320, 1 October 2026) composes 19 Linux apps into 1.2M annotated desktop observations. Fine-tuning Qwen3.5-4B on 200K examples raised ScreenSpot-v2's mobile split from 82.44 to 93.21 with no phone screens in the data, and long-horizon WebArena-Infinity tasks from 31 to 50 of 119. phoneshell's own crawler records 113 iPhone screens with centre points and no boxes or per-element effects.
PACE (arXiv:2610.02932, 2 October 2026) measures when turning a repeated computer-use task into a program pays: 11 of 20 compilations passed, payback ran 1.39 to 15.5 uses, and failed attempts used 67.7% of compile tokens. phoneshell skips compilation by recording sessions as macros at zero model cost, but saves them unverified, and its one stored macro has empty expect checks and one run in 785 audit events.
Zingrillo, Foerster, Shumailov, Zhao and Mullins (arXiv:2610.03089, 2 October 2026) show planner-isolated computer-use agents still fall to attacks that pick a pre-approved branch (89.5%), and cut that to 0 of 199 by checking destinations and parameters at an HTTP proxy. A phone agent has no such proxy. phoneshell's substitute, 14 label patterns on taps, misses Send, Unsubscribe and Erase All Content and Settings.
A vision web agent rose from 31.0 to 57.0 in the WebRetriever Challenge 2026 with the same claude-opus-4-6 throughout (arXiv:2610.03036, 2 October 2026), by fixing the code between model and browser. Checked stage by stage, phoneshell avoids the coordinate bug that cost WebFovea most, but its MCP server shrinks the iPhone screen to 235 x 512 and its no-change notice has no budget behind it.
Agents Are Systems, Not Models measures how much of an agent's score is just rolling again, and the answer is over half. We checked the one place our public iPhone data repeats itself. The flips are real but lopsided, and they trace to framework commits and a phone that kept leaving the app, an axis the paper's sandbox holds fixed.
ZoneClaw's point is that an agent's memory is where untrusted content goes to become trusted. We counted what phoneshell's screen memory holds on the rig: 808 screens, 16,815 labels, no provenance. Only 178 can reach a prompt today, and one of them carries a Wi-Fi network name, text anyone in radio range can choose.
ROGUE measures whether computer-use models yield when a person takes control back, and most do not. Run the same question against a phone agent and the answer comes from the framework, before any model is asked: phoneshell treats the iPhone's own stop gesture as a fault to heal in two of the three places that see it.
A new pymobiledevice3 command reads the iPhone's own frame counter: 0 means nothing on the display rendered for about a second. For a phone agent that is tempting as a settle signal. The release's own numbers say it is twice as slow as the screenshot gate phoneshell already uses, and blind on the screens that gate was built for.
Three xcuitest driver releases landed on 3 October 2026. The one that matters for real iPhones is 12.14.1: a tunnel registry that answered with an empty list used to win over a usbmux lookup that could see the phone. phoneshell does not run Appium, but its own bring-up makes the same mistake with usbmux, and tells the user to plug in a phone that is plugged in.
Three small mobile agents, two Android-style environments, one pool of their own experience split four ways. Where a button sits belongs in the weights; what a task usually does next, and what must be true before an action works, belong in the prompt. A prompt-only phone agent such as phoneshell can only use half of that, and currently stores less than half.
Six backbones, one optimizer that rewrites the code around them, and gains of 1.49 to 10.16 points on tasks it never saw. The two numbers that matter for anyone running their own evaluation: the same optimizer fed one run per task found less than half the gain, and 12 to 20% of tasks flipped between pass and fail across three runs of an unchanged agent.
cua-speedrun made the environment answer in milliseconds and the agent got slower: 89.54 to 99.02 seconds per task, with 40% more action batches, at the same score. A fast screenshot of a screen that has not redrawn is a wasted turn. phoneshell waits about 0.9 seconds before it looks, which avoids most of this and still loses the race to any app slower than that.
Absorbed in Inertia (arXiv:2609.37176, 29 Sep 2026): a 'screen did not change' notice moved repeated no-effect actions by -23.4% to +16.9% across four agents, while a full context reset changed the next action 96.4 to 100% of the time. phoneshell's live loop defence is exactly that notice; its StuckDetector class is never called.
SCOUT (arXiv:2609.36201, 28 Sep 2026): tool-use verifiers IRA and VAGEN score 38.9 and 29.4 unsafe F1 against 69.1 for a screenshot-only judge; SCOUT reaches 75.4 only by writing task rubrics in a separate tool-free step (53.1 without). phoneshell's 153 checks look only for what should be present; agents left 621 unwanted alarms that no check caught.
Neuro-Symbolic Computer Use (arXiv:2609.36927, 29 Sep 2026) turns one agent run into a refined script that beats the agent on Pass^3 in all four settings at 15 to 217x lower per-run cost, but on OSWorld its Pass^1 is below the agent on both models (53.6 vs 55.9, 72.9 vs 74.1). phoneshell's macros have no refinement loop, no outcome checks, and an unmeasured twenty-point claim.
Appium's WebDriverAgent shipped 16.13.0 to 16.13.6 on 30 Sep 2026. Per the pull requests' own simulator measurements, accessibility-id finds got 24.8 to 30.6% faster but a 1,029-node /source read only 3.31%, although the XML writer alone got 5.41x faster: the time is in the snapshot. phoneshell pins v16.12.4 and tries accessibility id first on every tap.
ActionEngine v2 (arXiv:2602.20502, 28 Sep 2026): the crawler-built map alone scores 73.1% on 655 WebArena tasks, 9.6 points under Claude Code on the same Claude Opus 4.6; the 91.2% headline follows a warm-up over one generated task per WebArena template, whose cost is not reported. phoneshell's crawler never types or toggles, so a crawl-built iOS map would lack exactly the form operations that warm-up repaired most.
LocalLSTC v2 (arXiv:2608.25777, 25 Sep 2026): with a Qwen3.5-9B planner, 78.6% of failed OSWorld runs stop without evidence of completion. Its Final Verification rejects a stop in 32.1% of tasks but removing it costs 2.9 points, inside two 2.7-point SDs; removing the kept subgoal costs 9.6. phoneshell ends every run at phone_done with no verifier.
Routide (arXiv:2609.29032) runs Qwen3.6-35B-A3B on an iPhone 17 Pro Max at 1.8 to 3.1 tokens/s in under 3 GiB, with a 0.00% LRU hit rate at 512 MiB explained by a 540 MiB per-forward working set. Prefill is one token per forward, so by our extrapolation phoneshell's 2,542-token tool list alone would take 13.9 to 23.6 minutes to read.
PhoneWorld v3 (arXiv:2605.29486, 28 Sep 2026): seven general agents average 86.0% easy and 83.3% hard on its generator-labelled mock-app tasks, against 86.0% and 31.6% on AndroidWorld. Its four-app audit finds 97.7% of real page visits rendered but 73.0% functional. phoneshell's own difficulty labels are hand-set and never checked against solve rates.
AgentTell (arXiv:2609.32915): six backbones picked the option revealing a secret they had just read in 61.1% of 7,630 sessions despite a privacy instruction, and in 34.5% of the 4,663 leaks (about 1,609) told the user nothing was disclosed; only the server log was right.
WebPageBench (arXiv:2609.35026, 28 Sep 2026) grades 24 model and framework pairs from the site's event log and keeps the agent's done claim separate: gpt-5.6-luna on OpenManus said done on 152 of 152 tasks with 90 verified, while OpenHands left 74 of 383 verified successes unreported across five models. The self-report error changes sign with the framework, so no single correction works.
GUI-Hopper (arXiv:2609.30887) and PhoneCLI (arXiv:2609.35671) both skip navigation with shortcut catalogs on Qwen3.7-Plus, and both peak at a middle catalog size: a full deeplink list cut untrained Qwen3.7-Plus from 61.5% to 53.0%, and a 100-screen map cut Settings from 95.7% to 78.3%. phoneshell's own open_url has no landing check beyond 'something moved'.
Jev-Mobile (arXiv:2609.30186) reports 79% on AndroidWorld vs 84% for a step-wise Qwen3.8-Max agent at 73.4% lower cost per success. Its step-wise baseline's summed executor cost and mean cost per success disagree 38-fold, Jev itself is 0.2% of spend, and the gap is 5 or 6 of 116 tasks from one run (Fisher p 0.39 to 0.50).
CAVEAT (arXiv:2609.27273) moved one worse product into a sponsored slot: visits rose from 45 to 60 of 60 runs, purchases from 0 to 48. Optimal purchases fell 78.6% to 17.3% under eight steering tactics; the harness ablations run on 15 episodes per arm, and a label-based spending gate like phoneshell's would approve every wrong purchase.
On MVCAP-Bench (arXiv:2609.27461), 13 GUI agent configurations scored 13.8% to 20.8% on a six-option motion CAPTCHA where chance is 16.7%; by exact binomial test 12 of 13 match chance, and seven backbones scored exactly 27 of 200 on one level, which independent guessing would do with probability 1.2e-10.
In a 325,000-run study (arXiv:2609.24927), Claude Opus 4.8 picked flights $248 dearer for a wealthy persona after reading two emails and $59 after reading all ten; 9 of 13 models peaked below full inbox access. Phone agents have no subject-only mode, and phoneshell's deny lists omit Mail and Messages.
Reconstructing paired flips from XYEval's Tables 1 and 16 shows Claude Opus 4.8 lost 15 Terminal-Bench tasks to misleading advice and gained 10 (McNemar p 0.42). Every binary cell gains tasks as well as losing them, which is one-run noise at temperature 1.0 that the paper never measured.
84.1% of 778 drive-by splash ads found in 31,823 Android apps fire on phone motion, and 28 of 130 defence and app combinations stopped the redirect; the only tool that always worked taps the skip button. phoneshell's exact-match close matcher, tested today, rejects 'Skip 5s', 'Skip Ad' and 跳过.
OSWorld-Pro's best judge, GPT-5.6-Sol at max reasoning, agrees with humans on step feasibility at 61.9 macro-F1 against 91.6 between humans, and labels 5.1% of steps infeasible against a human 15.3%. More reasoning made that column worse, which charges environment failures to agents.
ERPBench grades computer-use agents against the ERPNext database they leave behind. Commit-minus-database gaps computed from its Table 3 reach 82 points for UI-TARS-7B, which pressed Save in 85% of multi-field runs and stored the right record in 3%.
A static audit of the MCP server released with iOSWorld: of the 471 tools a model sees by default, 102 can write an app's state file directly and 65 read it, and send_zelle writes the payment itself because the app's Send button does not persist. The +12.0-point Table 7 ablation (Qwen3.5 12.8% to 24.8%) therefore changes four things at once.
iOSWorld's Appendix K reports 133 of 12,255 frontier swipes were the iOS back gesture and that 1,669 of 16,267 vision-only taps were a miss retried within 60 px, a pooled 10.26%. Both are rig problems: phoneshell already exposes back as a named action and taps by element rather than by coordinate.
Appium 4.0.0-beta.0 reached npm at 20:39:59 UTC on 19 September 2026 and moves human-readable CLI output from STDERR to STDOUT while tightening engines.node to a range that excludes Node 20, 21, 23 and 25. appium-xcuitest-driver has not shipped against it, so the gap is the window to fix the plumbing.
vivo's BlueLM-GUI scores 87.4 on its own MobileGUI-VBench and 84.9 on AndroidWorld. Counting the benchmark file shows 52 of 150 tasks graded on clarification or handover, capping an agent that never asks near 65.3. The report also shows a stricter judge added 4.9 points of RL success.
PriMobiBench (arXiv:2609.13873, 12 September 2026) says its mask cuts user profiling by up to 58% at about 8% task loss. Those come from different rows: the 8% configuration cuts GPT-5-mini's profiling by 32.1%. Every model saw screenshots only, so the accessibility-tree text an iPhone agent sends was never masked.
appium-xcuitest-driver 12.12.5 and 12.12.6 (18 and 19 September 2026) made UDID matching case-insensitive without changing the string. We tested why: xcodebuild, devicectl, pymobiledevice3, ideviceinfo and our own phoneshell all reject the lower-case form of an iPhone 17 Pro Max's UDID.
Since Xcode 27 shipped on 14 September 2026, a WebDriverAgent runner launched through devicectl can die with XCTest error 10300 on iOS 26 and iOS 18 phones. Appium fixed it on 20 September by launching through DVT with ActivateSuspended. The host's Xcode, not the phone's iOS, is the variable.
A head-to-head of mobile-agent evaluators puts the best LLM judge at 90.9% accuracy against human labels, with two humans agreeing 88.4%. On a 133-task benchmark that error is about 12 tasks while the model gaps being published are about 5.
Screen collection on a phone is a graph traversal, so phoneshell's crawler runs it in code at about 1.2 seconds a step; what makes it safe on a personal device is an 80-label refusal list, five type rules and a cap of three screens explored per layout.
iOS offers no second interactive surface for an agent: the external display scene role is named NonInteractive and iOS 27 stopped handing it to apps built against the new SDK, so phoneshell's background mode is a watchdog that yields the phone within one 0.8 second poll.
The 80-task phoneshell suite declares twelve check kinds and uses five: 76 of 80 tasks assert the foreground bundle id and seven assert nothing else, including three app-switching sequences whose single assertion is satisfied by their first leg.
Reading the outcome table of a three-week study of computer-use agents with blind users: model choice moved commands between failure and success, while partial completion stayed at about a third for all five models, concentrated in late-task termination and commit errors.
The iOS Auto-Lock page exposes no selection state; only a checkmark button inside the chosen row distinguishes it, so checking for the option's text proves nothing. The note documents three false-success checks found on 8 September 2026 and the witnessed routine that replaced them.
A batch of seven deletes from one accessibility read removed the owner's alarm, because iOS re-anchors a list's scroll offset when content shrinks. The note records five delete strategies that failed silently or did damage, the real switch geometry, and the one-delete-per-read rule.
After a cold boot a passcode-protected iPhone withholds developer services until a person enters the passcode, so WebDriverAgent cannot start and cannot type the stored passcode. A reboot ends an unattended session, and iOS updates and the 72 hour inactivity reboot cause the same state.
The fourteen overlay shapes a phone agent meets, published as data: the tell for each in the iOS accessibility tree, the moves that close it in order from safest to most destructive, and the never-tap rules. Plus why sheet detection must use vertical offset rather than size, and why the reflex belongs in the harness rather than the prompt.
iOS 26.6.1, released 17 August 2026 with 29 CVEs fixed, left a WebDriverAgent runner that launches, reports success, exits without a crash report and refuses connections on port 8100. Every health check passed. The runner is tied to the OS build, and the repair has a two-signing-team trap that reports IXUserPresentableErrorDomain error 1.
iOS Settings ships accessibility identifiers containing a UUID, a MAC address and a raw heap pointer. They change between app launches, so a screen fingerprint built on raw identifiers never accumulates. Also the measured cost of verifying an element's label against its pixels: 39% false alarms unscoped, zero over 46 elements when scoped to the six types whose label is the rendering.
An audit of every failure in one 32-task iPhone benchmark run: three failures were broken checks of our own, one was an unfair task that depended on the owner's device, and one or two were the model's. Alongside it, the five distinct ways the same rig produced a published number that was not true, and the guard now in code for each.
POST /session in WebDriverAgent begins by killing the active session, and the teardown terminates that session's tested application by default, so creating a session mid-task sends the iPhone to the home screen. A benchmark check that needed a session therefore destroyed the state it was measuring.
Measured on a physical iPhone on 8 September 2026: WebDriverAgent read, tap and drag latency scaled with accessibility tree size (154 ms to 24,195 ms for /source), a 640-row read silently dropped 218 of 631 switches, and tree depth 25 cost six times depth 20 for 19 more elements.
A moving W3C pointer chain costs about 19 seconds on a physical iPhone running iOS 26.6 regardless of how many points it contains, while WebDriverAgent's native drag endpoint performs the identical gesture in 1.4 seconds and a tap through the same W3C endpoint takes 500 ms.
A screen-by-screen diff of macOS Vision OCR against the iOS accessibility tree on a physical iPhone 17 Pro Max: on eleven of thirteen first-party screens the tree already carries every word OCR can read, and the only render-only text is on Maps tiles and Books cover artwork.
WebDriverAgent turns an iOS picker wheel with one tap at a fixed fraction of the wheel's height, and on the 292 point Clock timer wheel that default lands two rows away, making half the values unreachable. Agents that drag the wheel instead never converge: clock.timer.set was passed by one of six frontier models.
A measured catalogue of eight WebDriverAgent, devicectl and usbmux calls that report success on a physical iPhone while nothing happens, with the cost of each and the read-back that catches it. Measured on WebDriverAgent v16.12.4, iOS 26.6, 8 and 9 September 2026.
Blolabel's pre-registered screenshot ablation on a physical iPhone: 30/30 with the image, 0/30 without, and all 30 blind answers substituted the page's written word for the picture's word instead of abstaining. Read against iOSWorld (June 2026), where adding the accessibility tree gained Opus 4.6 25.6 points but cost GPT-5.4 Mini 10.5.
On Blolabel's 9 September 2026 six-model sweep, 16 of 27 device-graded failures carried a completion claim in the model's own summary; self-report grading would have reported 96.8% against the device's 92.3%. The error ran in one direction only, and three failures were rig outages that should not have been scored.
Adding 34 tasks to the benchmark took the ranking further out of reach, not closer. The reason is arithmetic that contains no sample size: McNemar needs six one-directional disagreements to reach p < 0.05, and our widest pair has had four since the first run.
No notes published yet.