iOSWorld's own arXiv abstract says 52% overall, 37% on multi-app; its own ICML listing for the same result says 51% and 36%

iOSWorld, a 133-task benchmark for iOS phone agents, reports its best configuration reaching "52% overall but only 37% on multi-app tasks" in the abstract posted to arXiv on 8 June 2026. The ICML 2026 virtual listing for the same paper, the same workshop poster, the same benchmark run, reports 51% and 36%. Nobody at either venue has to resolve this for the reader: both pages are public, both are citable, and they disagree by one point on both of the paper's two headline numbers.

What changed, and when

iOSWorld was posted to arXiv as 2606.09764 by six authors at Carnegie Mellon University (Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh and Ruslan Salakhutdinov), submitted 8 June 2026 with no later revision on record. It builds 26 new iOS apps around one persistent, seeded user identity, native to an iOS simulator, and sets 133 tasks across three bands: 27 single-app, 60 multi-app (spanning 2 to 8 apps), and 46 memory-and-personalization tasks that depend on data planted earlier in the same identity. The paper's stated purpose is to test whether agents can use a user's own cross-app history and preferences, not just operate one screen at a time.

The abstract at arxiv.org/abs/2606.09764 states the best configuration reaches "52% overall but only 37% on multi-app tasks," and that privileged vision+XML access, giving the model both a screenshot and the accessibility tree together, improves frontier models by up to 26 percentage points while smaller models see no benefit from the added tree. The paper's own listing at the ICML 2026 workshop "Agents in the Wild: Safety, Security, and Beyond" (icml.cc/virtual/2026/67944), same title, same six authors, states the best configuration reaches 51% overall and 36% on multi-app tasks, repeating the identical up-to-26-point accessibility-tree claim. Both pages were read on 25 October 2026. Neither page references the other's number, and neither flags a correction.

A benchmark phoneshell cannot check against either number

phoneshell's own board (site/results.json) runs 58 tasks per model against real, unmodified consumer apps on a physical iPhone 17 Pro Max, iOS 26.6, grading from WebDriverAgent state rather than a model's account of itself; across its six most recently run models the pass rate ranges from 89.7% (Qwen3-Max) to 96.6% (Kimi K3). iOSWorld's 52%-or-51% is not a number phoneshell's pass rate can be checked against: different apps (26 seeded synthetic apps built for the paper versus real third-party software), different task count and shape (133 tasks weighted toward multi-app and memory recall versus 58 single-pass tasks), and a different execution surface (an iOS simulator versus a physical phone). The only thing this piece is checking is whether iOSWorld's two public descriptions of its own result agree with each other, and they do not, by one point each time.

What it means for anyone citing a benchmark number

• A paper's own venues do not always restate its own headline number the same way. The gap here is one point, small enough that a reader citing either page alone would never notice it was ever anything else, which is exactly why it is worth checking both before repeating a figure in a comparison table like the one above it.
• The accessibility-tree gain claim, "up to 26 percentage points" on arXiv and "up to 26%" at ICML, is the one figure that travelled unchanged between both citations. Whatever produced the one-point drift on the other two numbers did not touch this one, which argues against a simple global rounding or rendering error and for something specific to how the overall and multi-app rates were recalculated or re-entered between the two postings.
• Neither page states whether its number reflects a later evaluation pass, a different scoring script, or a transcription step between the arXiv LaTeX source and the ICML listing's own copy. That is a real gap this check could not close: we read both numbers correctly, we could not find why they differ.
• For phoneshell's own purposes the practical lesson is narrower: any time a future note on this site cites iOSWorld's headline rate, it should name which of the two numbers it is using and link the specific page, rather than writing "iOSWorld reports 52% (or 51%)" as if the two were interchangeable rounding of the same measurement.

Check it yourself

# iOSWorld's own two public numbers for the same result
curl -s https://arxiv.org/abs/2606.09764 | grep -o "5[12]% overall but only 3[67]% on multi-app tasks"
curl -s https://icml.cc/virtual/2026/67944 | grep -o "5[12]%.\{0,40\}3[67]%"

# phoneshell's own board, for contrast, not comparison
git clone https://github.com/Blomega/phoneshell && cd phoneshell
python3 -c "import json; d=json.load(open('site/results.json')); print(d['board'])"

What would prove this wrong

We claim iOSWorld's arXiv abstract and its own ICML 2026 workshop listing report two different pairs of numbers, 52%/37% versus 51%/36%, for what both pages describe as the same best-configuration result. That is wrong if either page has since been edited to match the other, or if a camera-ready proceedings version states explicitly that the ICML figure reflects a later, corrected evaluation pass; we found no such statement on either page as of 25 October 2026. If a v2 of arXiv:2606.09764 appears and reconciles the two, this piece should be updated to say so rather than left standing as written.