iOSWorld's tool layer doubled Qwen3.5's score, and 102 of its 471 tools can write the app's data file directly

We read all 26 modules of the MCP server released with iOSWorld. The +12.0-point ablation changes four things at once, and only one of them is the interface.

iOSWorld's structured-tools ablation doubled Qwen3.5's pass rate, 12.8% to 24.8% over 133 tasks, and the released tool server that produced it is not an interface over the app. We read all 26 of its per-app modules: 102 of the 471 tools a model sees by default can write the app's data file directly, 65 more read the data file without touching the screen, and the Zelle tool says in its own docstring that the app's Send button does not save the payment, so the tool writes it instead.

What was published, and when

iOSWorld, arXiv:2606.09764, went up on 8 June 2026: 133 tasks over 26 purpose-built SwiftUI apps seeded for one fictional user. The code went up the same day as a single commit, e91f4cb at 2026-06-08T19:12:47Z in github.com/ljang0/iosworld, and has not been pushed to since. Appendix E is the ablation. It holds Qwen3.5 35B-A3B, the task set, the judge and the 50-step budget fixed and swaps the 7-action mobile_use tool for "a structured per-app tool layer" with operations such as caltrack.log_food and mybank.send_zelle. Table 7 reports the result:

The task counts are our arithmetic: 17 of 133 is 12.78% and 33 of 133 is 24.81%, the only integers that round to the printed rates. The paper reads the result as evidence that much of the vision-versus-tree gap is "interface rather than reasoning". The appendix does not say what the tools do underneath. The repository does.

What the 471 tools actually do

The server lives in mcps/: one module per app, 537 functions decorated @mcp.tool(). The runner hides prepare_* tools and confirm_* tools whose only argument is draft_id unless --with-confirmation-tools is set, which removes 66 and leaves 471 on the default surface. We parsed every module with Python's ast and followed each tool through the module-level helpers it calls. A tool is counted as writing state if any reachable path calls _data_layer.write_envelope, write_app_state, write_user_defaults or delete_user_defaults_key. The data layer's own header describes its purpose: "When a UI flow can't be cleanly driven via accessibility IDs, MCP tools can mutate the underlying state file (JSON / SQLite / UserDefaults) and then trigger an app reload".

Split by app, the direct paths are concentrated where the tasks are. Notes is referenced by 41 of the 133 tasks and 12 of its 15 default tools can write the notes file. CloudDocs, referenced by 35, has 12 of 21. Ten apps have no data-layer tool at all, including QuickChat, the most referenced app at 44 tasks, and Mail at 29.

Rows do not sum across because 26 tools matched neither pattern. Task reference counts are from the paper's Section 3. Every count in the table is ours, from a static pass, and it measures what a tool can reach, not how often Qwen called it; the per-call log the runner writes to artifacts/mcp_tool_calls.jsonl was not released with the paper.

One tool, end to end: send_zelle

mybank.send_zelle is the example the paper names, and task mybank-003 asks for exactly it: "Send $50 via Zelle to Maya Patel in MyBank and confirm the payment was sent", seven rubric criteria. The tool fills the form through Appium and taps zelle_send_button. Its comment then records a fact about the app: "The Send button closes the sheet but, in this build, does NOT persist a Zelle transaction (verified: state + shared ledger stay empty)." So it counts Zelle transactions before and after, and when the count has not moved it debits checking by writing the store directly. Its docstring adds that an unknown recipient "is still accepted and used verbatim".

The last box is the part that touches grading. _load_mcp_judge_trajectory in scripts/appium_agent.py hands the judge the MCP trace from trajectory.json, whose documented shape stores each call as {"action": "mcp_tool_call", "tool": ..., "result": {...}}, and the judge prompt builder in llm_action_generator.py writes json.dumps(actions) into every step line. In the pixel arms that string is a list of taps. In the MCP arm it contains the tool's own report of what it achieved. The screenshots still go in, and a data-file write followed by reload_app does change what the next screenshot shows, so the judge is not blind. It is reading the same evidence plus a sentence written by the thing being graded.

cloudsheets.edit_cell shows the other pattern. It taps the cell and the formula bar and types inside try: ... except Exception: pass, then sets the cell in the workspace envelope and reloads the app regardless of whether the typing landed. 49 of the 102 writers wrap their UI calls this way. For those tools a failed interaction and a successful one produce the same state and the same return value.

What it means if you are building or reading a phone-agent benchmark

The Table 7 comparison changes four things at once: action granularity, direct reads that return records the screen never displayed, direct writes that bypass the app, and outcome text in the judge's input. The paper attributes the +12.0 points to the first. The code does not let anyone separate them, because the ablation was run once, on one model (the README states MCP mode is Qwen-only and the runner rejects other models), and the per-tool call log is not public.

That does not make the gain fake. The paper's own showcase, Figure 7, is dinespot-001, 17 steps and score 1.0 with tools against 50 steps and 0.25 without, and DineSpot is one of the ten apps with no data-layer tool. Typed actions over the UI do help a small model. The judgement we would make is narrower: 24.8% is an upper bound on what a tool layer over the interface buys, and the gap to it is unmeasured.

For comparison, phoneshell's MCP server exposes 23 tools in phoneshell/mcp_server.py, and the benchmark runner allows 14 of them (AGENT_TOOLS in phoneshell/bench/runner.py). None is app-specific and none has a code path that writes an app's storage. Our 80 tasks carry 153 checks, and every one reads either the accessibility tree after the run or the agent's answer text. The agent's narration never reaches the grader. That design costs us exactly what iOSWorld's tool layer buys: a small model has no shortcut, and our numbers for small models will look worse for it.

If you want a tool-layer arm that means what it says, three changes are cheap. Remove the state-file fallbacks from any tool the model can call and let the call fail honestly. Strip result from the judge's copy of each action. Log every call and publish the log, so a reader can count how many of the 16 extra passes went through step 4.

Check it yourself

git clone --depth 1 https://github.com/ljang0/iosworld && cd iosworld
git log -1 --format='%h %aI'                      # e91f4cb 2026-06-08T19:12:47Z
grep -h "^@mcp.tool" mcps/*.py | wc -l            # 537
sed -n 1,10p mcps/_data_layer.py                   # "mutate the underlying state file"
grep -n "does NOT persist" mcps/mybank.py          # the send_zelle comment
grep -n "except Exception" -B8 mcps/cloudsheets.py | grep -n "tap_id\|type_text"
grep -n "json.dumps(actions)" scripts/llm_action_generator.py
python3 -c "import json;print([t for t in json.load(open('tasks.json')) if 'Zelle' in t['goal']][0]['name'])"

For the classification, walk each module's AST, collect functions whose decorator is mcp.tool, follow calls into module-level helpers, and flag any call on dl named write_envelope, write_app_state, write_user_defaults or delete_user_defaults_key. Drop names starting prepare_, and confirm_ tools whose only parameter is draft_id, to get the 471. The whole pass is about 40 lines and needs no simulator.

What would prove this wrong

The claim is that some of Table 7's +12.0 points is carried by paths that bypass the interface. It fails if a rerun with the state-file fallbacks disabled and result stripped from the judge input keeps Qwen3.5 at 24% or above on the 133 tasks. It also fails if the authors publish the call log and fewer than 3 of the 33 passing tasks invoked a state-writing tool. We will run the first version ourselves against the released code and publish the number, whichever way it lands, by 31 December 2026.