UI-TARS-7B pressed Save in 85% of ERPBench runs and stored the right record in 3%

ERPBench grades computer-use agents against the database they leave behind. Subtract its commit column from its database column and the gap is the whole story for open-weight models.

On ERPBench's multi-field tier, UI-TARS-7B pressed Save in 85% of runs and left the correct record in the database in 3%. The same model reached the target form in 95% of single-field runs and wrote the right value in 9%. The benchmark, arXiv:2609.17885, posted on 15 September 2026, is the cleanest published measurement of the gap between an agent committing and an agent being right.

What ERPBench measures

Kratika Bhagtani, Kusha Sridhar, Maziyar Baran Pouyan, Yuying Zhao and Eugene Siow built 30 tasks, 150 runs, on a live ERPNext instance and graded every run against the values in its database rather than against the final screenshot. Agents see pixels only and emit one action per turn: click, type, select, set date, scroll, key press or navigate, each tagged with a risk category (safe, commit or irreversible). The tiers are T1 single-field edits, T2 multi-field record creation, and T3 chains of 2 to 4 dependent documents such as Lead to Opportunity. The grader scores four ordered stages: Navigation (the record or field is reached), Interaction (the correct value is entered, T1 only), Commit (save or submit is invoked) and Database (the stored fields match ground truth).

The abstract's sentence, "some agents save in up to 85% of runs but write the correct value in as few as 3%", is one cell pair from Table 3. The rest of the table is more useful than the headline, so here it is with the subtraction done.

Values are the percentage of runs reaching each stage, from Table 3 of arXiv:2609.17885. The gap columns, commit minus database in percentage points, are ours. The paper does not publish per-tier run counts, so treat single-digit cells as a handful of runs.

The stages are not a funnel

You would expect each stage to be a subset of the one before. It is not. On T1, Qwen3-VL-32B enters the correct value in 30% of runs and commits in 42%; Holo3 enters it in 53% and commits in 47%; UI-TARS enters it in 30% and commits in 68%. An agent that saves more often than it types the right thing is saving forms it has not finished or has filled wrong. The paper's failure taxonomy names the two silent cases: Perception (a wrong value is saved) and Save-step (a correct value is never saved). UI-TARS-7B's failed T1 runs are 59% Perception and 27% Grounding. Qwen3-VL-32B's are 59% Grounding, 15% Planning, 15% Perception and 12% Save-step.

Humans are not immune either. Human #3 completed every T3 chain to the commit stage and left the correct records in 87% of them. That 13-point human gap is the useful calibration: some of what the database stage catches is task ambiguity rather than agent failure, and anyone copying this design should expect a nonzero floor.

What it means for a phone-agent grader

ERPBench is desktop and web, not iOS, and ERPNext is a system you can query. The lesson transfers anyway, because most phone-agent graders stop at a stage that looks like Commit. A check that the right app is in front is Navigation. A check that the confirm button was tapped, or that a toast appeared, is Commit. Neither sees the value that was stored.

Our own 80-task suite has 153 checks: 76 foreground_app, 31 element_exists, 23 regex_on_screen, 20 answer_contains and 3 element_exists_anywhere. The 76 are Navigation-level by construction. The 57 tree and regex checks read what the app renders after the run, which is one step closer to the store than a Save event and one step short of it: a list row can render a value the app failed to persist, and it would pass. We have already published that 7 of the 80 tasks assert only foreground_app. On a physical iPhone, first-party apps such as Clock give an external grader no database to query, so the rendered tree after a fresh relaunch is as close to the store as we can get without the app's cooperation. Relaunching before the read is the cheap step we do not take today, and this table is the argument for taking it.

Two practical rules follow. Grade the stored value, or the closest rendering of it after a cold relaunch, never the act of saving. And report the commit-to-correct gap next to the pass rate; a model that saves 85% of the time and is right 3% of the time is more dangerous to deploy than one that saves 5% of the time, and a single pass-rate column hides that.

Check it yourself

curl -sL https://arxiv.org/html/2609.17885 -o erp.html
python3 - <<'PY'
import re, html
x = open('erp.html').read()
figs = re.findall(r'<figure[^>]*ltx_table[^>]*>(.*?)</figure>', x, re.S)
for tr in re.findall(r'<tr.*?>(.*?)</tr>', figs[2], re.S):   # Table 3
    print(' | '.join(html.unescape(' '.join(re.sub(r'<[^>]+>', ' ', c).split()))
                     for c in re.findall(r'<t[dh].*?>(.*?)</t[dh]>', tr, re.S)))
PY
# gap = Commit minus Database, per tier; UI-TARS-7B T2 is 85 - 3 = 82

# our check mix, from the repository root
cat environments/*.yaml | grep -o "kind: *[a-z_]*" | sort | uniq -c | sort -rn

What would prove this wrong

The claim is that commit-level grading overstates agent success by a margin large enough to change rankings, and that the margin is concentrated in open-weight models. It is wrong if a rerun of the same six agents on ERPBench, or a second database-graded benchmark, shows commit-minus-database gaps under 10 points for every model on every tier. On our side, we will add a cold relaunch before every tree check on the 57 non-navigation checks and publish how many of the recorded passes flip. If none flip across a full three-model run, rendered-tree grading was already at database level for our task mix and this note overstates its relevance to phones. That count will be published by 31 December 2026.