Memory hallucination causes 58.9% of MemGUI-Bench's non-timeout failures; phoneshell's screen memory is re-checked against the live tree before every tap, so it cannot hallucinate the same way

MemGUI-Bench names five distinct memory-hallucination failure modes and finds they account for 58.9% of its 11 evaluated agents' non-timeout failures on average, with Agent-S2's multi-attempt success falling 20 points when its long-term memory is removed. phoneshell persists its own cross-run screen memory to disk, but by design never acts on it until the remembered element is found again in the live accessibility tree.

MemGUI-Bench (arXiv:2602.06075), first posted 3 February 2026 and last revised 27 August 2026 by Guangyi Liu, Pengxiang Zhao and 13 coauthors, argues that mobile GUI agent benchmarks have mostly ignored memory: across existing benchmarks, only 5.2 to 11.8% of tasks require remembering anything from earlier in a session or a prior one, and none test cross-session learning at all. Their benchmark corrects for that directly: 128 tasks across 26 apps, arranged as 64 mirror pairs, with 89.8% of all tasks requiring retention across time or across a different part of the app than where the needed fact was shown. Eleven agents across five architectures were run through it with a three-stage "Progressive Scrutiny" evaluation pipeline and seven hierarchical metrics covering memory fidelity, learning effectiveness, and execution efficiency.

Five named ways to misremember a screen

The paper's failure-mode taxonomy gives memory failure five separate names rather than treating it as one undifferentiated bucket: Partial Memory Hallucination (PMH), Process Memory Hallucination (ProcMH), Output Memory Hallucination (OMH), Knowledge Deficiency (KD), and Intent Misunderstanding (IM). Averaged across the 11 agents tested, memory hallucination (the PMH/ProcMH/OMH group) accounts for 58.9% of every failure that was not a plain timeout. The gap this produces against a memory-light benchmark is largest for the smaller, GUI-specialised models: Agent-S2 falls from 54.3% on AndroidWorld to 27.3% on MemGUI-Bench, GUI-Owl-7B falls from 66.4% to 6.2%, and UI-Venus-7B falls from 49.1% to 5.5%. The paper's clearest controlled test of what long-term memory is worth comes from a 40-task ablation subset (its Table 3): removing Agent-S2's long-term memory entirely drops its three-attempt success rate (pass@3) from 45.0% to 25.0%, a 20-point fall, holding everything else about the agent fixed.

phoneshell remembers differently, and it is not attempting MemGUI-Bench's task

phoneshell's (github.com/Blomega/phoneshell) own memory system, phoneshell/memory.py, is not ephemeral. It writes a JSON file per app bundle ID to a persistent store (STORE = RUNTIME / "memory"), loaded fresh at the start of a run through _load(bundle_id) and written back through save(bundle_id), so what one run learns about an app's screens is still there on the next run, on a different day, started by a different task. What gets stored is narrow by design: a structural fingerprint of a screen (hashed after stripping volatile text like prices, timestamps, and the UUIDs and MAC addresses iOS bakes into some accessibility identifiers), the location and label of its controls, which action led to which screen and how reliably, and how long that screen takes to settle. The module's own docstring states the rule that governs all of it: "Nothing here is trusted blindly: a remembered control is still resolved against the live tree before it is tapped, and a transition that stops holding decays."

That rule is the structural difference from the failure mode MemGUI-Bench documents. Partial, Process, and Output Memory Hallucination, by the paper's own framing, are failures of an agent acting on a RECALLED fact that turns out not to match the current state. phoneshell's design makes that category close to impossible to trigger by construction, because a remembered control is never the basis for an action; it is only a candidate that still has to be found and confirmed in the live tree before anything happens. The tradeoff is scope, not correctness: phoneshell's memory only ever stores procedural, navigational facts about how a screen behaves, never declarative facts about what a user said, preferred, or was told earlier in a conversation. None of phoneshell's 80 tasks ask an agent to recall a preference stated three screens ago or in a previous session, the exact shape of MemGUI-Bench's 89.8% memory-intensive task pool. phoneshell would not hallucinate on that task category. It has never been asked to attempt it.

What it means for anyone building agent memory into a phone agent

• The hallucination failure mode MemGUI-Bench documents is specifically a risk of acting on a model's generative recall of a fact. A memory system that stores structured, re-verifiable state (a fingerprint, a label, a coordinate) and requires a fresh match against ground truth before use is a different design, not a smaller version of the same one, and it is not eligible for the same failure.
• That design has a real cost: it cannot do what MemGUI-Bench actually measures, remembering a user's stated preference or an instruction given earlier, because it was never built to store that kind of fact at all. Adding declarative, cross-session preference memory to a tree-verified system like phoneshell's would require a second memory path, not an extension of the existing one.
• Agent-S2's 20-point pass@3 drop without long-term memory is the paper's most direct evidence that persistence itself, not just better single-shot reasoning, is doing real work. A system that resets its understanding of an app on every run is giving up a specific, measured amount of performance for the simplicity of starting clean.

Check it yourself

# phoneshell's own memory persistence and its re-verification rule
sed -n '1,30p' phoneshell/memory.py
grep -n "resolved against the live tree\|decays" phoneshell/memory.py

# confirm it persists across runs, keyed by app, not per-session
grep -n "STORE\|_load\|def save" phoneshell/memory.py

# MemGUI-Bench's own failure-mode taxonomy and tables
curl -s https://arxiv.org/html/2602.06075 | grep -i "hallucination"
# full tables: https://arxiv.org/html/2602.06075 (Table 2, Table 3, appendix A.8.1)

What would prove this wrong

We claim phoneshell's memory design structurally cannot produce Partial, Process, or Output Memory Hallucination because it never acts on a remembered fact without re-matching it against the live accessibility tree first. That claim is falsified if a future version of phoneshell/memory.py adds a code path that acts on a cached fact without a live re-check, or if an audit of a real run finds a tap executed against a stale, unverified memory record. We also claim phoneshell has no equivalent of MemGUI-Bench's cross-session preference or instruction recall. That is wrong the moment a phoneshell task asks an agent to recall something stated in an earlier, separate run and scores it. We will treat either event, checked by 31 March 2027, as grounds to revisit this comparison.

Sources

• Liu, Zhao, et al., MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents, arXiv:2602.06075, submitted 3 February 2026, revised 27 August 2026 (v3), read 16 October 2026.
• Full tables (Table 2 agent scores, Table 3 long-term memory ablation, Appendix A.8.1 failure-mode definitions): arxiv.org/html/2602.06075.
• Blolabel, phoneshell: phoneshell/memory.py, read 16 October 2026.