A sponsored slot raised agents' visits to a worse product by 15 runs in 60 and its purchases by 48

CAVEAT holds catalog, prices and the right answer fixed and switches on the tactics real marketplaces use. Agents mostly saw the worse product already; sponsorship changed whether they bought it. A spending gate that checks the action, not the choice, would have approved every one.

Moving one suboptimal product into a sponsored slot near the top of the results raised the share of agent runs that visited it from 45 of 60 to 60 of 60, and the share that bought it from 0 of 60 to 48 of 60. The agents already saw it in three runs out of four and never chose it; sponsorship changed the verdict, not the discovery. The numbers are from CAVEAT (arXiv:2609.27273, submitted 23 September 2026 by Yuxuan Li of Carnegie Mellon and Will Epperson, Wesley Deng and Zezhou Huang of Microsoft Research).

What CAVEAT measured

CAVEAT is nine browser marketplaces (lodging, retail, food delivery, resale, freelance services, grocery, specialty shops) in which every scored task has exactly one product that meets the user's hard requirements and is best on all stated preferences. Each task runs twice with the same catalog, prices and optimum: once as a matched control, once with eight steering mechanisms switched on (sponsored placement, preferential ranking, drip pricing, promotional framing, defaults and bundling, scarcity and social proof, trust signals, friction). Every mechanism shows only true information. Agents run in the BrowserUse framework; the main sweep is 52 tasks, 3 repetitions, 5 model configurations.

Across that sweep the optimal purchase rate fell from 78.6% to 17.3%, which on 780 episodes per condition is about 613 against 135 (our reconstruction; the paper gives percentages). On CAVEAT-Hard, five tasks with 2,112 products over 88 result pages, GPT-5.6-Sol at high reasoning bought the optimum in 90.0% of controls and in none of 50 steered episodes. All 50 failures stopped on page 1.

Counts are ours, rebuilt from the percentages and the stated designs: CAVEAT-Shop is 20 tasks, and the CAVEAT-Harness comparisons use three repetitions per task, so 60 episodes on Shop and 15 on Hard. Every percentage in Section 6.1 lands exactly on those denominators (7/60 is 11.7%, 12/15 is 80.0%). Intervals are Wilson 95%.

Where steering enters: three points, each with a number

The placement experiment is the cleanest of the three. Across five product types and four preference settings, 60 runs per arm, the request, catalog and attributes are fixed and only the position of one suboptimal product changes. Visits rose by 15 runs; purchases rose by 48. Once it was sponsored, 80% of the runs that saw it bought it, against 0% of the 45 that saw it unsponsored. On the evidence side, the 23 runs that dropped a found optimum came from 19 model, task and condition settings, and 13 of those settings had another repetition that bought the optimum once the decisive fact was read correctly.

Three of the paper's claims are smaller than they read

The abstract says CAVEAT-Harness "raises user-optimal purchasing by 55.0%". That is 55.0 percentage points, 11.7% to 66.7% for GPT-5.6-Terra on Shop, or 7 to 40 of 60 episodes: a 5.7-fold rise. The relative figure is larger than the abstract says, and the unit is wrong.

The Hard ablation of CAVEAT-Harness runs on 15 episodes per arm. Verification alone reached 10 of 15 and the full version 12 of 15; Fisher's exact p for that difference is 0.68. On this sample the structured objective adds nothing measurable, and the paper's reading that verification "accounts for most of the gain" is as far as the data go.

The transfer evidence is 12 episodes per arm on four CAVEAT-Stay tasks. CAVEAT-Harness took GPT-5.6-Terra from 2 to 6 of 12 (p 0.19); CAVEAT-27B reached 2 of 12 against 0 of 12 for its backbone (p 0.48). The paper calls both "preliminary", which is accurate: neither is distinguishable from no transfer. The in-domain post-training result does hold: 11 of 48 against 2 of 48 on the held-out Shop tasks, p 0.014.

What it means for an agent buying on a phone

phoneshell's spending rail confirms that a purchase happens, not which product. The 14 confirm_patterns in phoneshell/config.py match labels such as "place order", "buy", "checkout" and "subscribe"; max_spend_local is 0.0, so every payment needs a human tap; and rule 6 of the MCP instructions tells the model to show the user what it is about to do before calling again with confirmed=true. In CAVEAT's placement arm, all 48 wrong purchases would have reached that gate as ordinary purchases of a real product at a true price. A human approving "buy item X for $Y" sees what the agent saw. None of the 14 patterns compares the chosen item with anything else on the screen.

CAVEAT's strongest single lever was a check on the choice, not the action: a tool the agent can call before buying to argue that its search was complete and that no unresolved fact could change the comparison. For a phone agent that means the confirmation message should carry the evidence (how many results were inspected, whether the price shown is the payable one) rather than only the item name. That costs time; the paper measured 2.6 times the runtime. A separate study of default and social-proof nudges on a test web store (arXiv:2609.19843, 17 September 2026) points at the same gap from the defaults side.

Check it yourself

• Paper: https://arxiv.org/abs/2609.27273; HTML https://arxiv.org/html/2609.27273v1. Placement numbers are in Section 5.2, drip pricing in 5.3, single-mechanism ablation in 4.1, CAVEAT-Harness arms in 6.1.
• Intervals and tests: from scipy.stats import fisher_exact; fisher_exact([[12,3],[10,5]]) gives 0.68; fisher_exact([[2,10],[6,6]]) gives 0.19; fisher_exact([[0,60],[48,12]]) gives 3.2e-22.
• Denominators: python3 -c "print(7/60, 40/60, 18/60, 12/15, 10/15, 2/15, 1/15)" reproduces every Section 6.1 percentage.
• Our gate: sed -n 125,150p phoneshell/config.py and grep -n confirmed phoneshell/mcp_server.py in the phoneshell repo.

What would prove this wrong

Prediction, dated 31 March 2027: in any rerun of the CAVEAT-Hard ablation with at least 30 episodes per arm, verification alone lands within 15 points of the full CAVEAT-Harness. If the full version beats verification alone by more than 15 points at that size, the structured objective matters and our reading of the 15-episode arms was too cautious. Second prediction, same date: a CAVEAT-Stay transfer test with at least 60 episodes per arm shows the CAVEAT-Harness gain as significant (p below 0.05); if it does not, CAVEAT-Harness is a Shop-specific fix.