MobileJailBench's adaptive attacker raises Gemini-2.5-Pro's jailbreak rate from 50.7% to 83.6% in one round; phoneshell's only hard block is three credential app IDs, never attacked

A static jailbreak test tells you how a mobile agent does against last year's attacks. MobileJailBench instead distils each failed attempt into a stronger one and runs it again, and that single adaptation round adds 32.9 points to Gemini-2.5-Pro's attack success rate. phoneshell has never run anything adversarial against its own safety code at all.

MobileJailBench, posted to arXiv as 2610.23303 on 16 October 2026 by Songping Wang, Borui Li, Qinglong Liu, Yueming Lyu, Yi Huang, Fengting Li, Chao Feng and Caifeng Shan, describes itself as the first evolving jailbreak benchmark for mobile agents: 520 tasks across three jailbreak paradigms and 14 risk categories, run in a mobile sandbox of 28 apps, 14 third-party apps spanning communication, finance, social media, commerce, navigation and entertainment, plus 14 system apps including Messages, Settings, Contacts, Calendar, Notes, Gallery, Browser and File Manager. The paper's headline claim is a 100% attack success rate (ASR) on Qwen3.5-122B-A10B reached through what it calls Dual-Dynamic Evolution. Its own Table 3(a), read directly, shows that same model at 98.6% under its evolved attack, a 1.4-point gap between the abstract's round number and the table it is presumably drawn from that the paper does not reconcile anywhere we could find.

What changed, and when

The mechanism behind the gain is Dual-Dynamic Evolution, which pairs two processes. Failure-Distilled Jailbreak Evolution (FDJE) takes a prompt that failed to jailbreak a model and rewrites it using what the model's own response revealed about why it refused, then retries. Weakness-Driven Benchmark Expansion (WDBE) does the complementary thing at the benchmark level: once FDJE has found a reliable new attack pattern, WDBE mints new tasks that generalize it, so the benchmark's own task pool grows around each model's demonstrated weak points rather than staying fixed at 520 tasks forever. The paper reports a second round of this expansion, WDBE-50, a set of 50 new tasks built from exactly this process, and re-ran it against the same ten models: first-try success on the new tasks alone ranged from 36.0% (Gemini-2.5-Pro) to 96.0% (Qwen3.5-122B-A10B), confirming the expanded tasks were not simply easier versions of the original 520.

What makes this different from a conventional jailbreak suite is not any individual number, it is that the benchmark does not hold still. A static red-team set, scored once, measures resistance to the attacks someone thought of before publication. MobileJailBench's FDJE loop means the attack a model is scored against on day one is not the attack it will be scored against on day thirty; every refusal teaches the benchmark something about how to ask again.

The baseline number undercounts every model it was tried on

Table 3(a) in the paper reports the same ten models twice: once against a static pass-at-2 baseline, and once after Failure-Distilled Jailbreak Evolution has had a single round to adapt. Every model's attack success rate rises. Gemini-2.5-Pro moves from 50.7% to 83.6%, a 32.9-point gain, the largest in the table. Gemini-3-Flash moves from 65.0% to 87.9%. Gemini-3.1-Pro moves from 62.9% to 88.6%. Qwen3.5-397B-A17B, already at 86.4% static, still gains 10.7 points to reach 97.1%. The smallest gain in the table belongs to Kimi-K3, which rises only 3.6 points, from 60.0% to 63.6%, the one model in the set the adaptive attacker struggled to move.

The paper's scoring method is built to separate two things a single pass/fail number conflates: whether a model said it would refuse, and whether it then acted on the harmful instruction anyway. Its Stratified Adjudication Framework (SAF) assigns each episode a label on three axes, security awareness (did the agent express a safety-motivated refusal before acting), execution behavior (did it adopt the prohibited objective and act on it), and risk consequences (did a state oracle confirm the prohibited outcome actually happened on the device). From those three labels it derives a Reasoning Hijacking Rate (RHR), the share of episodes where a model's own refusal language appears in its output while it executes the harmful action anyway. The paper validates SAF against a reference set and reports 97.26% agreement and 98.1% stability, against 91.14% and 95.7% for a single LLM acting as judge; on RHR specifically, SAF matches the reference rate of 83.3% where the single judge undercounts it at 63.3%, a 20-point miss on the exact failure mode, an agent narrating safety while its hands do something else, that a pass/fail grader would not see at all.

What it means for anyone grading a phone agent on safety

• A pass/fail jailbreak score hides the exact failure MobileJailBench was built to surface. phoneshell's Guard never records whether a tap it flagged for confirmation was preceded by anything resembling the model reasoning about risk, because Guard.classify_tap takes only the tap's own label text, never the model's chain of thought; there is no channel for an RHR-style check to run through even if someone wanted to build one.
• phoneshell's one hard block, the three-bundle-ID denylist in classify_app, covers a narrower slice of risk than almost any other mechanism in this comparison: it stops an agent from opening Apple's own Passwords or Keychain apps, and nothing else. MobileJailBench's 14 risk categories run across finance, commerce, social and navigation apps that phoneshell's denylist has no mechanism to ever name, confirm, or refuse.
• A benchmark that evolves against a model's own failures, the way FDJE and WDBE do, answers a different question than a fixed task set can: not "does this model resist the attacks we thought of," but "does this model's resistance generalize past the first prompt that worked." phoneshell's confirm_patterns list has had no equivalent exercise; nothing has ever tried to find the phrasing that gets past it, and so whatever score that list would earn on an adaptive attacker is simply unmeasured.
• The 1.4-point gap between MobileJailBench's abstract (100% ASR on Qwen3.5-122B-A10B) and its own Table 3(a) (98.6% under FDJE) is small, but it is the kind of inconsistency this site's house rule is built to flag rather than smooth over: the paper's round number and its own table do not agree, and neither we nor, as far as we can tell, the paper itself resolves which one is the intended headline figure.

Check it yourself

# MobileJailBench's own numbers
curl -s https://arxiv.org/abs/2610.23303 | grep -i "attack success\|520 tasks\|risk categories"
# full text with Table 3(a), 3(b), 3(c): https://arxiv.org/html/2610.23303

git clone https://github.com/Blomega/phoneshell && cd phoneshell
cat phoneshell/safety.py                         # Guard.classify_tap always returns allowed=True
grep -n "denied_bundle_ids\|confirm_patterns" phoneshell/config.py   # the 3-bundle denylist and the 14 fixed patterns
grep -rn "jailbreak\|adversarial\|red.?team" phoneshell/ || echo "no adversarial testing in this repository"

What would prove this wrong

We claim phoneshell's Guard class has never been tested against an adversarial prompt of any kind, static or adaptive. That is wrong the moment any commit adds a test file, script, or log that runs a jailbreak-style prompt against classify_tap or classify_app and records the result; we found none as of 23 October 2026. We also report the 1.4-point discrepancy between MobileJailBench's abstract and its Table 3(a) as unresolved in the paper; that is wrong if a later revision of arXiv:2610.23303 reconciles the two figures, which we will check by 30 April 2027.