How well does Jev-Mobile do on AndroidWorld?

Jev-Mobile (arXiv:2609.30186, 24 September 2026) reports 0.79 task success on the full AndroidWorld suite, against 0.84 for a step-wise Qwen3.8-Max agent and 0.78 for SeeAct-V with UI-TARS-1.5-7B grounding. On 116 tasks the 5-point gap is 5 or 6 tasks from one run, and Fisher's exact test gives p 0.39 to 0.50.

Where does Jev-Mobile's 73.4% cost saving come from?

Jev's own charges are $0.013091 across about 92 successes, $0.00014 each, 0.2% of Jev-Mobile's $0.072744 per success. The saving comes from calling the Qwen planner about 3.8 times less per successful task, not from a cheap executor.

Is the Jev-Mobile cost table internally consistent?

Not for the step-wise baseline. It uses one model, so its summed executor cost over successes should equal its mean cost per success times its successes. The table gives $0.694774 against about $26.55 (0.273694 x 97), a 38-fold difference.

Can an accessibility-tree executor tap something that is only in the pixels?

Not in Jev-Mobile: it has no coordinate fallback, so a target with no tree node returns BLOCKED. The paper promises coverage, selection and handoff failure counts but does not report them. phoneshell's MCP tools make the same choice: none of its 23 tools takes a coordinate.

← Notes

analysis 27 September 2026 6 min read

Jev-Mobile's 73% cost cut rests on a table whose baseline cost rows disagree 38-fold

A typed decision model picking accessibility-tree IDs got within 5 points of a step-wise VLM on AndroidWorld at a quarter of the spend. The 5 points are 5 or 6 tasks from one run, the savings are averaged over different tasks, and one baseline row cannot be reconciled with another.

Jev-Mobile, posted to arXiv on 24 September 2026, reports 79% on AndroidWorld against 84% for a step-wise Qwen3.8-Max agent, at $0.072744 of model spend per successful task against $0.273694, a 73.4% saving. Its own Table 1 does not reconcile: for the step-wise baseline, a one-model system, the summed executor cost over successful runs is $0.694774, while the mean cost per success implies about $26.55, 38 times more. And the 5-point success gap is 5 or 6 of AndroidWorld's 116 tasks from a single run, which no test on these numbers can separate from zero.

Abstract technical illustration of a large slow node sending a single beam to a small fast node that fans out into many short steps along a grid of labelled-looking boxes, one box left unreached

What Jev-Mobile changed, and what it reported

Linghua Zhang, writing from a Rice University address, splits a mobile agent into two speeds. A vision-language model, Qwen3.8-Max (qwen/qwen3.8-max-0902 on OpenRouter), reads the task, the screenshot and the accessibility tree, and emits a local goal plus any exact text to type. A program then enumerates every visible, enabled node in the fresh tree as a numbered candidate (click, long-press, scroll, focus, plus Back, Home, Enter and Open app), and Jev, a typed decision model, picks one ID or returns DONE or BLOCKED. After each action the tree is re-read and the IDs are rebuilt. Jev never sees the screenshot, and there is no coordinate fallback: a target missing from the tree cannot be tapped.

The comparison has three systems on the full AndroidWorld suite: step-wise VLM (Qwen chooses every action), SeeAct-V (Qwen decides, UI-TARS-1.5-7B grounds, substituted for the original UGround), and Jev-Mobile. Success uses every task instance. Every time and cost row uses only each system's own successful trajectories.

Metric	Step-wise VLM	SeeAct-V	Jev-Mobile	Source
Full-task success rate	0.84	0.78	0.79	Table 1
Successes out of 116 (ours, from the rate)	97 or 98	90 or 91	92	AndroidWorld task count; arithmetic ours
Mean total time per success (s)	197.21	162.63	132.67	Table 1
Mean executor time per success (s)	94.30	12.37	5.16	Table 1
Time outside the executor (s, ours)	102.91	150.26	127.51	Table 1, subtraction ours
Executor cost, summed over successes (USD)	0.694774	0.030579	0.013091	Table 1
Mean all-model cost per success (USD)	0.273694	0.193207	0.072744	Table 1
Executor cost per success (USD, ours)	0.00716	0.00034	0.00014	Table 1 divided by successes, ours
Coverage, selection and handoff failure counts	not reported	not reported	not reported	promised in Section 1, absent
Number of runs per task	not reported	not reported	not reported	paper

The paper never states how many task instances it ran. AndroidWorld ships 116 tasks (Rawles et al., arXiv:2405.14573), and the reported rates fit that: 92/116 is 0.793, 97/116 is 0.836 and 98/116 is 0.845. If the authors ran a different count, the task arithmetic below moves with it; the cost inconsistency does not depend on it.

The step-wise baseline's two cost rows disagree 38-fold

Section 5.2 defines executor cost as "the sum of that model's API charges over successful trajectories", and the executor for step-wise VLM is Qwen, which is also the only model that system calls. So its summed executor cost and its mean all-model cost times its successes are the same quantity. They are not: $0.273694 × 97 is $26.55, and the table says $0.694774, a ratio of 38.2. Either the executor row counts only part of Qwen's charges, or the mean row is inflated. We cannot tell which, and the headline 73.4% saving is computed from the mean row.

The two rows that do reconcile say something the abstract does not. Jev's own charges come to $0.013091 across roughly 92 successes, $0.00014 each, which is 0.2% of Jev-Mobile's $0.072744 per success. The other 99.8% is Qwen planning. The saving therefore comes from calling Qwen about 3.8 times less per successful task ($0.0726 against $0.2737), not from a cheap executor. Any model that could pick from the candidate list without being called at every step would earn most of it, which is the ablation the paper lists as untested (Jev replaced by a small VLM).

The success gap is five or six tasks, and the time and cost rows compare different tasks

At 116 tasks, 0.84 against 0.79 is 97 or 98 successes against 92. Treated as independent samples, Fisher's exact test gives p 0.39 to 0.50. The design is paired (same tasks), which is stronger, but the paper publishes no per-task outcomes. The most favourable case for a real difference is that every disagreement runs one way: 5 discordant tasks give an exact McNemar p of 0.0625, 6 give 0.031. Any task Jev-Mobile solves and step-wise does not pushes it back out of reach. The same arithmetic applies to "close to SeeAct-V": one task apart.

The efficiency rows compare means over different task sets. Step-wise VLM's average includes the 5 or 6 tasks that Jev-Mobile failed, and a task one agent solves and another does not is plausibly a harder, longer one. That is a judgement, not a measurement; the direction of the bias is what matters. Cost per attempted task, including failures that ran to the budget, is the comparison a buyer needs, and it is not reported.

Time tells a similar story once split. Step-wise VLM spends 94.30 of its 197.21 seconds (47.8%) in the executor; Jev-Mobile spends 5.16 of 132.67 (3.9%). Outside the executor, which for Jev-Mobile includes Qwen planning and for step-wise VLM is device interaction and waits, Jev-Mobile takes longer: 127.51 seconds against 102.91.

What it means for anyone acting on a phone by element ID

The design is the one phoneshell already ships. Rule 2 of the instructions our MCP server hands the model reads: "Act by id (phone_tap, phone_type). Never guess coordinates; if what you want is not in the table, scroll or re-observe." None of the 23 tools in phoneshell/mcp_server.py takes a coordinate. So the number Jev-Mobile left out is the one we would most want from it: how many of the failures were targets with no tree node. Our house result is that when an answer exists only as pixels, a screenshot takes the task from 0% to 100%. Jev-Mobile keeps the screenshot for its planner, so it can read such an answer and pass it as text, but it cannot tap such a target. Reading and acting fail differently, and one success rate hides which one happened.

Three decisions follow. Budget from cost per attempted task, not per success; an agent that fails expensively looks cheap on the paper's metric. Treat a 5-point gap on 116 tasks from one run as unresolved. And if you split planner and executor, count how often the executor returns BLOCKED for lack of a candidate; that is the ceiling the split imposes.

Check it yourself

Paper: https://arxiv.org/abs/2609.30186, HTML at https://arxiv.org/html/2609.30186v1; Table 1 is in Section 6, the cost definitions in Section 5.2.
Task count: AndroidWorld's 116 tasks, https://arxiv.org/abs/2405.14573.
The reconciliation: python3 -c "print(0.273694*97, 0.273694*97/0.694774)" prints 26.548 and 38.2.
The Jev share: python3 -c "print(0.013091/92/0.072744)" prints 0.00196.
The tests: from scipy.stats import fisher_exact, binomtest; fisher_exact([[97,19],[92,24]]); binomtest(6,6,0.5) give p 0.499 and 0.031.
Our side: grep -n "Never guess coordinates" phoneshell/mcp_server.py and grep -c "^def phone_" phoneshell/mcp_server.py in the phoneshell repo.

What would prove this wrong

If the author publishes per-task outcomes by 31 December 2026, we predict two things: the step-wise and Jev-Mobile runs will disagree on at least 10 tasks, with an exact McNemar p above 0.05, and one of the two step-wise cost rows in Table 1 will be corrected, the summed executor cost towards roughly $26 or the per-success mean towards roughly $0.007. If the per-task data show 6 or fewer disagreements all in step-wise's favour, the gap is real and we were wrong to call it unresolved. If both cost rows are confirmed as printed, we misread the definitions in Section 5.2 and will say so on this page.