ActionEngine's crawled map alone scores 73.1% on WebArena, 9.6 points under Claude Code; its 91.2% comes after a warm-up over the task templates

The map-then-program web agent matches Claude Code on three of four WebArena domains with its crawled map, trails it by 34 points on GitLab, and reaches its headline only after a template-driven warm-up whose cost the paper does not report.

ActionEngine's headline is 91.2% on 655 WebArena tasks, 8.5 points above Claude Code on the same Claude Opus 4.6 backbone. With only the map its crawler built, it scores 73.1%, which is 9.6 points below Claude Code and 34.0 below it on GitLab. The rest comes from a warm-up that runs one generated task for every WebArena task template and patches the map, and version 2 of the paper (arXiv:2602.20502, 28 September 2026) does not report what that warm-up costs.

What the paper changed and what it measures

ActionEngine is from Hongbin Zhong and Kexin Rong (Georgia Tech) with Fazle Faisal, Luis França, Tanakorn Leesatapornwongsa, Adriana Szekeres and Suman Nath (Microsoft). Version 1 was posted on 24 February 2026 and version 2 on 28 September. A crawling agent explores a web application offline and writes a state-machine graph: page templates as states, each with the operations available on it and the transitions they cause, serialised as YAML. At run time a planner reads that graph and writes a complete program in one call, which then executes without further planning. When a step breaks, a reactive fallback repairs it and a Patcher updates the graph.

The evaluation adds a warm-up between those two phases. The authors generate and execute one task per WebArena task template with the Patcher switched on, then switch it off for the scored runs, so each scored task is a single attempt on a fixed map. Warm-up tasks share templates with the evaluation tasks but not their instances or parameters. Failed tasks from every system were hand-reviewed, and answers that were correct but formatted differently from the reference were counted as passes.

Success rates are percentages. Break-even is the paper's: tasks needed for ActionEngine's lower per-task cost to repay the crawl, relative to Claude Code. It does not include the warm-up, whose cost is not reported. The paper links no code repository that we could find.

The crawl gets you to parity on simple sites, and the warm-up does the rest

On three domains the crawled map alone lands within 3.4 points of Claude Code: 89.0 against 92.4 on Reddit, 80.7 against 79.7 on Shopping, 74.0 against 74.2 on Shopping Admin. GitLab is the exception, at 55.0 against 89.0. The warm-up closes that and more: GitLab's map gains 66 operations and reaches 89.0, and Shopping Admin goes from 74.0 to 95.3 while losing 4 operations to merges. The authors sort the repairs into three kinds: selectors that did not fire the page's JavaScript handlers, missing waits after asynchronous updates, and operation descriptions that needed behaviour learned by running them, such as a save that stays on the edit page.

The efficiency numbers are large and hold after warm-up: 27 seconds per task against 87 for Claude Code and 787 for AgentOccam, 1.5 to 3.5 model calls against 7 to 16, and $0.05 per task against $0.40, of which prefix caching accounts for the step from $0.16 to $0.05. The success comparison is the part that depends on the warm-up. Our judgement is that "task-independent" fits the crawl but not the 91.2%, because the warm-up is steered by the benchmark's own templates. Nobody deploying this on a new site has a template list.

What it means for a phone agent with a memory

phoneshell has both halves of this design, built separately. phoneshell/crawl.py walks an iPhone's screens with no model in the loop at about 1.2 seconds a step, identifying screens by a layout fingerprint plus a content signature, and it never types, never toggles a switch and refuses 80 labels (send, pay, delete, allow, sign out and so on). phoneshell/memory.py records, during ordinary agent runs, which control on which screen led to which screen and how often. It treats a screen as known after two visits and shows the model up to eight remembered routes as a hint. Its docstring says a route walked enough times "becomes replayable outright, with no model in the loop"; in the code, memory only produces hints, and replay lives in the separate macro recorder.

ActionEngine's split of repairs maps onto that code. One of its three repair kinds, waits after asynchronous updates, phoneshell already handles by measurement: each remembered screen keeps a running mean of how long it took to settle and uses 1.4 times that, clamped to 0.15 to 2.5 seconds. The other two do not arise the same way on iOS, because phoneshell resolves controls against the live accessibility tree rather than stored selectors, and it has no operation descriptions to enrich.

The gap is the one the GitLab row shows. The crawled map was weakest where tasks go through forms, and the warm-up gains were largest on the two form-heavy domains, plus 21.3 on Shopping Admin and plus 34.0 on GitLab. A phoneshell crawl leaves out exactly the operations those forms need, by design, since a crawler that types and toggles on a real phone changes the user's data. Our judgement: on iOS a map-then-program agent would have to learn its input operations from supervised agent runs, the way memory.py already learns transitions. It could not get them from a crawl, and until it does, its crawl-only number is the one to expect.

The plan-then-execute limit also carries over. After warm-up, ActionEngine's remaining failures were mostly ambiguous intent (about 23 cases) and search-dependent tasks (about 22), where the planner has to pick search terms before seeing results. On a phone, search fields are the common entry point to Maps, Contacts, Settings and the App Store, all of which appear in phoneshell's 80 tasks.

Check it yourself

• Paper: https://arxiv.org/abs/2602.20502v2; Table 1 (success and latency by domain), Table 2(a) (crawl cost, time, break-even, states, operations, crawler-only success), Table 2(b) (prefix caching), Section 5.1 (warm-up protocol and manual review), Section 5.3.
• Arithmetic: crawler-only overall = (106 x 89.0 + 187 x 80.7 + 182 x 74.0 + 180 x 55.0) / 655 = 73.1; Claude Code minus crawler-only = 82.7 minus 73.1 = 9.6.
• phoneshell: sed -n 1,40p phoneshell/crawl.py (design notes, 1.2 s a step); grep -n "def suggested_settle" -A 4 phoneshell/memory.py; grep -n "def hint" -A 30 phoneshell/memory.py (known after 2 visits, 8 routes).

What would prove this wrong

Dated 31 March 2027. If ActionEngine, or a replication, runs the warm-up on half of WebArena's templates and scores only tasks from the other half, overall success will be below 85%. A held-out score at or above 88% would show the warm-up teaches reusable site knowledge rather than template coverage, and the 91.2% would stand as a deployment number. If a published follow-up reports warm-up cost, break-even including it will exceed 100 tasks on at least one of the four domains.