RiOSWorld finds agents intend a risky OS operation 91.3% of the time when induced; phoneshell's own patch notes describe a route to the same class of action that skips the agent, and the model, entirely

RiOSWorld induces ten multimodal agents toward risky goals on a real desktop and measures how often they intend, then complete, the risky action: 91.3% intention and 84.0% completion averaged across OS-operation tasks, 41.1% and 33.3% for file writes. Every one of those numbers describes a model deciding to do something unsafe. phoneshell's own code documents a different channel to the same class of harm, writing a file, that requires no model, no reasoning and no decision at all: an unauthenticated route, reachable by anything on the same wifi network, which is exactly why phoneshell prefers a cable and warns about the alternative in its own comments.

RiOSWorld (arXiv:2506.00618, Jingyi Yang, Shuai Shao, Dongrui Liu and Jing Shao, submitted 31 May 2025, revised to v3 20 June 2025, listed as a NeurIPS 2025 poster) asks whether safety behavior built into dialogue models survives the move to computer-use agents with real OS access. Across 492 tasks split between user-originated risks (238 tasks: Web, Social Media, Office, File I/O, OS Operation, Code, Multimedia) and environmental risks (254 tasks: phishing pages, phishing email, pop-ups, reCAPTCHA, account fraud, induced text), the paper reports an unsafe rate for both whether the agent intended the risky outcome and whether it actually completed it. The category closest to what an iOS automation harness touches, OS Operation, averages 91.3% intention and 84.0% completion across the ten models tested; File I/O, writing to disk, averages 41.1% and 33.3%. phoneshell's own code, read directly, documents a route to exactly that category of action that does not pass through any of what RiOSWorld is measuring, because no agent decision is involved.

What RiOSWorld measures: a model's own decision to do harm

RiOSWorld runs on a Linux-based virtual machine, screenshot-only observation at 1920x1080, agent actions issued through pyautogui, building on OSWorld's environment rather than a phone. Each risky task is modeled as a partially observable Markov decision process, and every trajectory is scored on two separate questions: did the agent's stated or inferred goal match the risky one it was induced toward (intention), and did it actually carry that goal out (completion). The gap between the two numbers in every category the paper reports is the share of cases where a model meant to do something unsafe but did not manage to, not where it recognized the risk and declined.

Source: arXiv:2506.00618, Tables 2, 3 and 4, read 28 October 2026. Averages are across ten tested models (GPT-4o, GPT-4o-mini, GPT-4.1, Gemini-2.0-pro, Gemini-2.5-pro-exp, Claude-3.5-Sonnet, Claude-3.7-Sonnet, Qwen2-VL-72B, Qwen2.5-VL-72B, Llama-3.2-90B-Vision), reconstructed from the paper's HTML tables; the paper's single overall per-model ranking appears only in a figure we could not read as text.

The route that skips the question entirely

phoneshell/wda/patches.py adds two custom routes to WebDriverAgent that Appium's own build does not ship, and its module docstring states plainly why: there is no other way to put a picture into the camera roll from automation, because the photo library is a database guarded by PHPhotoLibrary, not a directory a file write can reach, and WebDriverAgent is the only process already running on the phone that can call it. The docstring also measures the alternative it closes off: writing straight to /var/mobile/Media/DCIM over AFC lands a file on disk, confirmed with stat, while the photo library's own count stays unchanged across a write and a relaunch of Photos. Both new routes, importMedia and pressButton without a session, are declared .withoutSession, the same flag WebDriverAgent itself uses for its sessionless read routes. The file's own comment states the consequence without softening it: "anything that can reach the WebDriverAgent port can use them, and WebDriverAgent authenticates nothing."

phoneshell's own bring-up code already treats this as a real cost, not a hypothetical one. bringup.py prefers a cable connection over wifi specifically because the cable "does not put an unauthenticated automation server on the wifi," and the WebDriverAgent runner is launched bound to 127.0.0.1 only when the transport is not wifi; the moment phoneshell is driving a phone over wifi instead of a cable, that loopback bind is dropped. RiOSWorld's 91.3% intention rate on OS Operation and 41.1% on File I/O describe a model choosing, under inducement, to do something unsafe. The route patches.py documents requires no choice: a POST to /wda/customMedia or whatever path the patch registers, from any device on the same network as a phone running phoneshell over wifi, writes a file to that phone's camera roll with no model, no screenshot, no reasoning step, and no induced goal anywhere in the chain.

What it means for a harness that runs on a real phone

• These are different risk models, not competing measurements of the same thing. RiOSWorld studies whether an agent's own judgment fails under inducement; the gap in phoneshell's own code is an access-control question that never reaches judgment at all. Citing one as evidence about the other would overstate both.
• The mitigation that exists is a default, not a control. Binding WebDriverAgent to loopback when not using wifi is a choice launch_wda makes automatically, and bringup.py's own comment states the reasoning in plain terms, but nothing in patches.py itself adds a credential, a token or any check on the caller. The security property phoneshell currently has rests entirely on choosing the cable.
• RiOSWorld's File I/O category, the one closest in kind to importMedia, is the single lowest unsafe rate the paper reports among user-originated risks, 41.1% intention, which the paper's own framing would read as models being comparatively reluctant to write files under inducement. phoneshell's unauthenticated route removes the one actor, the model, whose reluctance that number is measuring.

Check it yourself

# the two custom, unauthenticated routes and their own stated risk
grep -n "withoutSession\|authenticates nothing" -B5 phoneshell/wda/patches.py

# the cable-over-wifi preference, stated in phoneshell's own bring-up code
grep -n "unauthenticated automation server" -B3 -A3 phoneshell/bringup.py

# the loopback bind, and when it is dropped
grep -n "bind_loopback\|127.0.0.1" phoneshell/bringup.py

# RiOSWorld's own category tables
curl -s https://arxiv.org/html/2506.00618v3 | grep -iE "OS Operation|File I/O|Unsafe"

What would prove this wrong

We claim phoneshell's custom WebDriverAgent routes carry no authentication beyond the default of binding to loopback when the transport is not wifi, so a phone deliberately or accidentally driven over wifi exposes an unauthenticated write path to its camera roll to anything on that network. That is wrong if a later version of patches.py or the WebDriverAgent fork it patches adds a token, a pairing step or a network check we did not find. We also claim this gap is architecturally distinct from the risk RiOSWorld measures, since it requires no agent and no induced goal. That distinction fails if someone demonstrates the route cannot actually be reached without a valid WDA session despite the .withoutSession declaration, which would mean our reading of the flag is wrong rather than the code. We will treat either finding, reproduced against a real device, as grounds to revisit this piece by 30 April 2027.