LearnAct turns one demonstration into a 32.4-point gain on a task the model has never seen; phoneshell's memory only helps on a screen it has seen before

LearnAct's KnowSeeker embeds a new task's instruction and retrieves the most similar stored demonstration by cosine similarity, before the agent has ever attempted anything resembling that exact screen. phoneshell's memory.py, examined for a different mechanism elsewhere on this site, solves a narrower problem than DeltaReplay's per-step routing: it caches a structural fingerprint of a screen and reuses what it learned only once that same fingerprint reappears. A task's first occurrence gets nothing from it, no matter how close a cousin it is to one already solved.

LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark (arXiv:2504.13805, Liu, Zhao, Liu, Chen, Chai, Ren, Wang, He, Meng, submitted 18 April 2025) attacks a specific gap in mobile GUI agents: a base model can be shown exactly one worked example of a task family and still fail to generalize, because the example and the new task differ in wording, screen state, or target content. LearnAct is a three-part pipeline. DemoParser converts a raw demonstration, instruction, coordinate-based actions, and screenshots, into a natural-language knowledge entry per step, templated as "[On/In] [Screen Name], [Action Details], to [Purpose]" for intermediate steps and "[On/In] [Screen], complete task, [Reason/Answer]" for the terminal one, with an optional "Memory" annotation carrying task-relevant facts a later step might need, such as a price read off an earlier screen. KnowSeeker embeds every stored instruction in advance with the all-MiniLM-L6-v2 sentence transformer; for a new task it embeds that task's instruction, ranks stored entries by cosine similarity, and returns the top matches above a threshold. Retrieval runs on instruction similarity alone, not on screen layout or action similarity. ActExecutor then runs the task with the retrieved knowledge folded into its prompt.

What one demonstration is worth, by model and setting

Table 4 of the paper reports offline action-match accuracy on LearnGUI-Offline, built by extending the AMEX dataset with standardized actions and k-shot combinations: Gemini-1.5-Pro rises from a 19.3% zero-shot baseline to 51.7% with one retrieved demonstration, a 32.4-point gain, and continues to 57.7% at three shots. Models that already score well with no demonstration gain far less in absolute terms: UI-TARS-7B-SFT moves from 77.5% to 82.8%, a 5.3-point gain. Table 6 reports online task success rate on LearnGUI-Online, built on AndroidWorld with 101 tasks: UI-TARS-7B-SFT rises from 18.1% to 32.8%, a 14.7-point gain, and Qwen2-VL-7B rises from 9.9% to 21.1%, a 11.2-point gain. The paper excluded an online LearnAct result for Gemini-1.5-Pro itself, citing limited task completion in preliminary tests, so the model with the largest offline gain has no reported online figure.

Source: arXiv:2504.13805v1, Table 4 (offline) and Table 6 (online), read 30 October 2026.

What phoneshell's memory.py activates on, and what it does not

phoneshell's own memory.py states its trigger condition in its module docstring without ambiguity: "Next time the same screen appears, the agent is told what it is and what worked here before." What is stored per screen is "a structural fingerprint of the screen, insensitive to the text that changes (prices, restaurant names, timestamps) but sensitive to the layout that does not," computed and matched against the current screen before any lookup succeeds. This is a different mechanism from agent/macros.py's named-macro replay, which this site covered separately in connection with DeltaReplay's similarity router: macros are invoked by an exact name a caller already knows, while memory.py's fingerprint lookup runs automatically on every screen, with no name required. But the two share the same structural limit LearnAct's design does not have. Memory.py's match key is the screen itself, computed after the agent has already landed on it; LearnAct's match key is the task instruction, compared before the agent has taken a single action. A phoneshell task that has never been run before, however similar in wording or intent to one already in the memory store, produces a cache miss on its very first screen and gets no benefit until, and unless, that specific screen reappears on a later run.

What it means for a memory system meant to help on new tasks

• LearnAct's gains are concentrated exactly where phoneshell's memory cannot reach: a model's first encounter with a task. A system that wants that benefit needs a second index, keyed on task semantics rather than screen structure, checked before the agent acts rather than after it lands somewhere.
• phoneshell's design is not a weaker version of LearnAct's, it solves a problem LearnAct does not: repeated execution of the identical task on the identical app build, where a fingerprint match lets the agent skip the model entirely rather than retrieve a hint for it. memory.py's own measured step costs, quoted in its docstring, show the win is "not asking the model at all," a stronger claim than LearnAct makes for any of its reported numbers, all of which still run the base model with retrieved context rather than skipping it.
• LearnAct's own ceiling is visible in its numbers: three-shot retrieval only lifts Gemini-1.5-Pro to 57.7%, six points past its one-shot score, so stacking more retrieved demonstrations has rapidly diminishing returns in this paper's own data, not a path to arbitrarily high accuracy.

Check it yourself

# phoneshell memory.py's own stated trigger: the SAME screen reappearing
sed -n '1,24p' phoneshell/memory.py

# confirm the fingerprint is computed from the screen, not the task instruction
grep -n "fingerprint\|def.*hash\|_VOLATILE" phoneshell/memory.py

# confirm there is no instruction-embedding or similarity search anywhere in memory.py
grep -n "embed\|cosine\|sentence.transformer\|similarity" phoneshell/memory.py || echo "no match"

# LearnAct's own description of KnowSeeker's retrieval method
curl -s https://arxiv.org/html/2504.13805v1 | grep -iE "KnowSeeker|cosine|MiniLM"

# LearnAct's Table 4 and Table 6 results, as quoted above
curl -s https://arxiv.org/html/2504.13805v1 | grep -iE "Table 4|Table 6|Gemini-1.5-Pro"

What would prove this wrong

We claim phoneshell's memory.py has no code path that compares a new task's instruction against stored knowledge before the agent reaches a screen, only a post-hoc structural fingerprint of the screen itself; that is checkable directly against the file and would be false the moment an instruction-embedding or similarity-search step is added upstream of any screen read. We also claim LearnAct's reported gains, up to 32.4 points offline and 14.7 online, come from retrieval keyed on instruction similarity specifically, which rests on the paper's own ablation design; a later version isolating KnowSeeker's contribution from DemoParser's knowledge format would sharpen or undercut how much of the gain is retrieval versus representation. We will revisit this piece if phoneshell adds any pre-action, instruction-level retrieval to its memory system, or if a later LearnAct revision changes the attribution between its three components, by 30 April 2027.