Thirteen GUI agents scored 13.8% to 20.8% on a six-option motion CAPTCHA, where chance is 16.7%

MVCAP hides the answer in motion, so every frame is noise. Twelve of thirteen agent configurations are indistinguishable from guessing, seven report the identical 27 of 200 on one level, and sampling frames at 10 per second barely moved them.

On MVCAP-Bench, a six-option CAPTCHA whose answer is visible only as motion, 13 GUI agent configurations scored between 13.8% and 20.8% against a chance level of 16.7%, while 30 human participants scored 99.6%. By our binomial tests on the 600 instances, 12 of the 13 are indistinguishable from guessing; the 13th, GPT-5.4 with native computer use at 125 of 600, clears chance at p 0.0072, which does not survive correction for the 23 configurations the paper reports. The benchmark is arXiv:2609.27461, submitted 23 September 2026 by Zeyu Zhang, Guangtao Zhai and colleagues.

What MVCAP changed

MVCAP renders the target as a field of dots that moves coherently against a background of dots that moves differently. Any single frame is dots. Three levels, 200 instances each: L1 text shapes, L2 rotating 3D objects, L3 point-light human actions. Each instance is a looping video in a live web page with a prompt and six answer buttons. A matched control, MVCAP-Bench-FG, removes the moving background and keeps everything else. Ten backbones run in Browser Use; GPT-5.4, Claude Sonnet 4.6 and Gemini 3 Flash also run as native computer-use agents; an offline setting feeds the same videos as frames sampled at 10 per second. Every instance is attempted once (Pass@1).

Correct counts are the average times 600, rounded; p values are two-sided exact binomial tests against one in six. Across 23 configurations (13 in the browser, 10 offline) the Bonferroni threshold is 0.0022. The two lowest p values, GPT-5.4 native at 125 of 600 and Gemini 3 Flash offline at 125 of 600, are both 0.0072.

Seven backbones scored exactly 27 of 200 on the text level

On L1, seven of the ten Browser Use backbones report 13.5%: GPT-5-mini, Claude Sonnet 4.6, Gemini 3 Flash, Gemini 3.1 Pro, GLM-4.6V, Kimi K2.5 and Qwen3.5-Plus. That is 27 correct out of 200 each. A model guessing uniformly lands on exactly 27 with probability 0.038, so seven independent guessers doing it together has a probability near 1 in 8 billion (1.2 × 10^-10). Four of the same models also report 13.5% on L2. The answers are not independent guesses. Our reading, which the paper does not test, is that models that cannot see the target fall back on the same option, whether by position or by plausibility, and 27 of the 200 L1 answer keys sit on that option. If so, the full-benchmark scores measure the answer key's layout, and a model ranking on them is noise.

The foreground control shows the models can read the targets once they are exposed: seven backbones reach 96.0% to 100.0% on L1 text with the background removed. Averages rise by 44.9 points for GPT-5.4 (16.8% to 61.7%), 54.2 for Gemini 3.1 Pro (13.8% to 68.0%) and 47.2 for Gemini 3 Flash (15.8% to 63.0%). L3 stays hard even then: the best foreground-only result is 29.5%.

More frames did not rescue it

Removing the browser and handing the model frames sampled at 10 per second raised full-benchmark averages from 13.8%-16.8% to 17.0%-20.9%. Eleven of the 60 offline cells are not multiples of 0.5 (for example 18.1%, 15.6% and 33.2%), so some offline denominators are below 200 and are not reported. The finding holds either way: time-sampled frames did not turn the motion into a readable signal for any model tested.

Money did not buy anything either. GPT-5.4 in Browser Use spent $57.06 on the full benchmark for 101 correct answers, where a die would expect 100; Claude Opus 4.6 spent $59.96 for 92; Qwen3.5-Plus spent $1.52 for 89.

What it means for phone agents and for the people blocking them

Our house result is that when an answer exists only as pixels, the screenshot is the whole task: 100% with it, 0% without. MVCAP is the next case down, an answer that exists only across pixels over time, and there a single image is worth nothing too. phoneshell's phone_observe returns an element table plus one picture, and none of its 23 tools records video. Anything on an iPhone that means something only in motion (an animated progress cue, a loading shimmer that implies state, a moving-dot challenge) is invisible to it by construction. We have not measured how often that occurs in real apps; the claim is about what the observation can contain, not about frequency.

For anyone deploying a challenge like this, the useful number is not 16.8%, it is the floor. A six-option format hands every non-seeing agent 16.7% free, and our reading of the 27-of-200 cluster says models may concentrate on one option, which a defender can exploit or be hurt by depending on the answer layout. Two consecutive passes drop a pure guesser to 2.8%. Randomise answer position per instance and check the rate at which each option is chosen, not just the accuracy.

Check it yourself

• Paper: https://arxiv.org/abs/2609.27461; HTML https://arxiv.org/html/2609.27461v1. Table 2 holds per-level Pass@1 and cost; Table 3 the offline results; Section 5.1 the human protocol (30 participants, 60 instances per dataset).
• The chance tests: from scipy.stats import binomtest; binomtest(125,600,1/6).pvalue gives 0.0072; binomtest(101,600,1/6).pvalue gives 0.91.
• The L1 cluster: from scipy.stats import binom; binom.pmf(27,200,1/6)**7 gives 1.2e-10.
• No code or dataset link appears in the paper as of 27 September 2026, so the answer-key layout cannot yet be checked.
• Our side: grep -n "def phone_observe" -A 12 phoneshell/mcp_server.py in the phoneshell repo.

What would prove this wrong

Two predictions, dated 31 March 2027. First, when the MVCAP instances or per-answer logs are released, the seven 13.5% L1 scores will trace to one shared answer choice (one option position or one option label) chosen in most of each model's L1 attempts. If the logs show those models spreading their answers across all six options, the identical scores were coincidence and our reading is wrong. Second, no agent configuration will exceed 33.3%, twice chance, on full MVCAP-Bench L1 under the paper's protocol by that date.