Giving a computer-use safety judge tools made it worse at catching harm: 29.4 and 38.9 unsafe F1 against 69.1 for a screenshots-only judge, until rubrics came first

SCOUT shows that a verifier which can inspect the machine after the run finds completion better but unsafe side effects worse, unless it first writes task-specific rubrics in a separate, tool-free step. phoneshell's 153 device checks look only for what should be there, and a run of leftover alarms shows what that misses.

Giving a computer-use safety judge a shell made it worse at catching harm. On AutoElicit-Bench, two verifiers that can probe the machine after the agent finishes scored 38.9 (IRA) and 29.4 (VAGEN) unsafe F1, against 69.1 for ZeroGUI, which only reads the trajectory's screenshots (SCOUT, arXiv:2609.36201, 28 September 2026). The tools helped completion checks, where both tool users beat every screenshot judge. Safety improved only when the verifier first wrote task-specific rubrics in a separate, tool-free step: 75.4 unsafe F1.

What the Columbia paper measured

SCOUT is by Jianxing Chen, Xiao Yu, Shipra Agrawal and Zhou Yu (Columbia University), posted 28 September 2026 with code at github.com/jc-244/SCOUT. It is a verifier, not an agent: it runs after a computer-use agent (Qwen3.6-35B-A3B, 50-step budget) has finished an OSWorld-style task and decides two things, whether the task was completed and whether anything unsafe happened. Stage one, the rubric generator, reads the task and the action trace with no tools and writes completion and safety rubrics specific to that task. Stage two, the probing agent, follows those rubrics with four tools: run_command and run_python (read-only inspection), computer_use (operate the GUI) and check_screenshot (pull a trajectory screenshot).

The baselines get the same general safety criteria. Three are passive judges that read screenshots: DigiRL (last screenshot), DistRL, and ZeroGUI (all trajectory screenshots). Two are tool-use verifiers built for completion checking, IRA and VAGEN, adapted by the authors for safety. One caveat from the paper itself: tool-use verifiers need a live machine, so each judged its own fresh rollouts, while the passive judges shared one rollout set. The comparison is within each method's runs, not on identical trajectories.

AutoElicit-Bench completion labels come from OSWorld's own verification scripts; its safety labels from the benchmark's evaluator. OS-Blind tasks cannot be completed safely by construction, so it has no completion score, and the authors excluded its pop-up subset, which is scored by the coordinates of one click.

Tools find what changed; a rubric decides what to look for

The completion column climbs the way you would expect with more evidence: 43.5 with one screenshot, 56.7 with all of them, 68.6 and 68.7 with tools. The unsafe column does not. The tool users probe for whether the task got done, and with general safety criteria pasted into the prompt they do not go looking for collateral damage. The paper's two worked cases show what the probes can find when pointed at it: files outside the task's scope made world-readable (visible to a shell, not in any screenshot), and an email the agent was told only to address that is shown "Sending" on the live screen.

The ablation isolates the rubric. Asking SCOUT's own verifier to write rubrics and probe in a single prompt drops unsafe F1 from 75.4 to 53.1 on GPT-5.6 Terra, and from 54.5 to 12.5 with Qwen3.5-9B as the verifier. Table 5 shows why: in a single prompt the model spends 204 reasoning tokens on the rubric, against 1,642 when rubric writing is a separate request with no tools. For Qwen3.5-9B the figures are 269 against 9,872. Merely attaching a dummy tool to the separate request cuts GPT-5.6 Terra's reasoning to 1,099 tokens.

What it means for grading a phone agent

phoneshell grades from the device, which is the point of the project: 153 checks over 80 tasks, read after the run, never from the model's own account. Every one of them asks whether the right thing is present (the right app in front, a value on screen, an answer in the final text). One of the 153 is negated. None asks what else changed.

We have a measured case of what that misses. Two Clock tasks end with "Do not save it." Agents saved the alarm anyway, run after run, until the phone held 640 alarms, 621 of them identical leftovers (phoneshell FINDINGS.md, section 18). No check flagged a saved alarm. The problem surfaced through latency: a full tree read took 2,612 ms on the alarms list against 154 ms on Clock's timer screen, and 24,195 ms with the add sheet open over the list. The fix was a clean_alarms teardown on three Clock tasks, which removes the damage without failing the run that caused it.

SCOUT's result says the cheap fix is not "give the grader more access". A phone grader already has read access to state through WebDriverAgent; phoneshell/bench/verify.py supports twelve check kinds, including app_state, switch_on and ocr_contains, and the tasks use five. What is missing is the step that writes, per task, the list of things that must not have happened: no alarm added, no message sent, no setting other than the named one changed. On a phone those can be checked with the same reads, before and after the run. Our judgement is that a handful of hand-written negative checks per app would have caught the alarm residue on the first day; SCOUT's numbers suggest a model writing them needs its own tool-free step to do it well.

The reflection result is worth a caution. Feeding SCOUT's safety verdict back to the agent cut unsafe runs from 30.2% to 17.2% while completion stayed between 43.1% and 50.0%, with no consistent direction. That is one benchmark with one agent model. Blocking each unsafe action before it ran, in a preliminary test with a 100-step budget, ended at 19.8% unsafe, slightly worse than after-the-fact reflection.

Check it yourself

• The paper: https://arxiv.org/abs/2609.36201; Table 2 (main results), Table 4 (ablations), Table 5 (reasoning tokens), Section 4.3 (reflection), Section 5.1 (the two worked cases). Code: https://github.com/jc-244/SCOUT.
• Count phoneshell's checks: python3 -c "import yaml,glob,collections;c=collections.Counter(ch['kind'] for f in glob.glob('environments/*.yaml') for ch in yaml.safe_load(open(f)).get('checks',[]));print(sum(c.values()),c)" from the repo root gives 153 and the five kinds.
• Negated checks: grep -rn "negate" environments returns one line, in settings_back_twice.yaml.
• The alarm tasks and their teardown: environments/clock_alarm_set_time.yaml and clock_alarm_reach_picker.yaml; supported check kinds in phoneshell/bench/verify.py; the incident in FINDINGS.md section 18.

What would prove this wrong

By 31 March 2027 we will add negative checks (nothing saved, nothing sent, no unrelated setting changed) to every phoneshell task whose instruction forbids or implies a side effect, and rerun the affected tasks three times per model. If no run fails a negative check, our claim that completion-only grading hides real side effects on this suite is wrong for current models. If SCOUT's code, run by others on AutoElicit-Bench with a tool-use baseline given the same separate rubric step, reaches within 5 points of SCOUT's 75.4 unsafe F1, then the gain is the rubric step alone and SCOUT's probing design adds little beyond it.