Rollback-Induced Reflection rolls an agent back up to 1.3 times a task; phoneshell has no checkpoint to roll back to
A method published 16 September 2026 restores a long-horizon agent to an earlier state and keeps what the failed attempt learned. phoneshell's only answer to a task that goes wrong is to let it run out of steps and call that a failure, with nothing kept and nowhere to go back to.
Rollback-Induced Reflection (RIR), submitted to arXiv on 16 September 2026 by Yi Yu, Liuyi Yao, Yaliang Li, Enshu Wang and Libing Wu, beats the strongest baseline on three long-horizon agent benchmarks by 3.09 to 6.57 percentage points, averaged per backbone model, by restoring execution to a chosen earlier state when a trajectory goes wrong and carrying forward what that failed attempt learned. phoneshell (github.com/Blomega/phoneshell), the harness that runs this site's benchmark, has no equivalent of any kind. Its only reaction to an agent that cannot finish a task is error_max_turns: a hard stop at a fixed step count, scored as an outright failure, with no saved state and no path back to one.
What RIR actually decides, and when
The paper frames recovery as a "rollback-boundary control problem": three decisions, not one. When to intervene. Where to resume. What to keep. Most prior methods pick one side of this and lose on the other, the authors argue: fixing the context without repairing the environment's actual state leaves the agent reasoning about a world that no longer matches what it sees, while restoring an earlier state the plain way throws away everything the failed attempt was worth. RIR's trigger condition is explicit in the paper's own notation: a review happens when the agent calls a rollback tool, or when a review horizon L_t has elapsed since the last review, whichever comes first (ReviewEvent_t = I[a_t = rollback] ∨ I[t − t_rev ≥ L_t]), with a default horizon of 4 to 8 steps. Calling the tool does not roll anything back by itself; it only requests a review. An LLM reviewer then returns a verdict, and a rollback executes only if that verdict is "Rollback" and the method's own rollback budget for the episode has not run out.
Where it resumes is the second decision. Rollback depth is defined as k = t − r, the distance from the current step back to the restore point r. Rather than always jumping back a fixed number of steps, RIR's restore rule looks for the latest checkpoint inside a localized interval that removes the condition that caused the failure, so a shallow mistake gets a shallow rollback. What survives the rollback is the third decision: not nothing, and not the full trajectory either. A "structured reflection memory" distilled from the abandoned attempt is carried forward to guide the next try, which is the paper's actual innovation over plain state-restoration: on ScienceWorld, RIR executes an average of about 0.9 to 1.3 rollbacks per task, each one keeping a piece of what the discarded path learned rather than erasing it.
phoneshell's only exit is down
phoneshell's step budget is a single config field: max_steps: int = 60 in phoneshell/config.py, labelled in the code's own comment as "Hard cap on agent steps per task, and on money." Each task file can set a lower per-task ceiling through Task.max_steps in phoneshell/bench/schema.py, which defaults to 25. runner.py hands that number straight to the model under test, either as max_steps on the OpenRouter path or as --max-turns on the Claude Code CLI path. There is no reviewer, no horizon, and no budget for a second chance: once the count is reached, the CLI exits 0 with {"subtype":"error_max_turns","is_error":true} and, per the code's own comment, "usually no result text." runner.py treats that exactly like a wall-clock timeout, setting timed_out = True, and the comment beside that line states the policy outright: "A model that runs out of time has FAILED the task... scoring it from whatever the phone happens to look like afterwards credits work it never finished." Nothing is salvaged. No earlier screen state is kept to return to, because none was ever recorded as a checkpoint in the first place; the only persisted record of the attempt is the row in results.json, which keeps the outcome (passed, turns, seconds, checks) and nothing about what the agent tried along the way.
| Dimension | Rollback-Induced Reflection | phoneshell | Source |
|---|---|---|---|
| What a budget overrun means | a reviewable event, not an automatic failure | an automatic failure (error_max_turns → timed_out) | arXiv:2609.18304 §3; runner.py |
| State kept from a bad attempt | structured reflection memory, distilled | none; only the pass/fail row survives | arXiv:2609.18304 §3; results.json schema |
| Where execution resumes | latest checkpoint that removes the failure condition (depth k = t−r) | nowhere; the task ends | arXiv:2609.18304 §3; bench/runner.py |
| Decision authority | a second LLM call (the reviewer) judges Continue vs Rollback | none; the step count alone decides | arXiv:2609.18304 §3; config.py max_steps |
| Typical intervention rate | ~0.9-1.3 rollbacks per task (ScienceWorld) | not applicable, no intervention exists | arXiv:2609.18304 §4 |
The gain is real, but it is not free, and it is not mobile
RIR's reported edge holds across three different model backbones on three benchmarks that have nothing to do with phones: ALFWorld (embodied household interaction), ScienceWorld (text-based scientific experimentation) and GAIA (open-domain tool use). Averaged per backbone, RIR beats the strongest baseline by 3.09 points on Qwen3-14B, 6.57 points on DeepSeek-V3, and 3.04 points on Qwen3.8-27B. Those are real, model-specific, benchmark-specific gains, not a universal multiplier, and none of the three backbones are the GUI-grounded models phoneshell evaluates (the OpenRouter and Claude Code CLI paths it drives today). Nothing in the paper claims a GUI or mobile result; it is cross-listed only to cs.CL and cs.RO.
What it means for anyone grading a phone agent
- phoneshell's choice to score a budget overrun as a flat failure is deliberate, not an oversight: the comment in
runner.pyexplains it is there to stop a harness fault from being read as a model capability, by refusing to credit a task state the model never finished reaching. That is the right call for a benchmark whose product is a trustworthy pass/fail number. - It is also a different question from whether an agent COULD have recovered. RIR's result says a reviewable rollback, not a hard stop, is worth 3 to 6.6 points on three unrelated long-horizon benchmarks. Nothing in phoneshell's current task set measures whether that would hold on a phone, because nothing in phoneshell ever gives an agent the chance to try.
- A rollback boundary presupposes a checkpoint to roll back to. phoneshell's
Snapshotobjects inactions.pyexist to compare screens before and after one action, not to serve as restore points; none of them are retained once an action completes, so there is currently nothing in the codebase a rollback mechanism could restore to even if one were added. - The review horizon RIR defaults to, 4 to 8 steps, sits well inside phoneshell's own per-task budget of 25 to 60 steps, so there would be room for several review cycles within a single phoneshell task before the hard cap is reached, if such a mechanism existed.
Check it yourself
# phoneshell's step budget and its only failure path
grep -n "max_steps: int = 60" phoneshell/config.py
grep -n "max_steps: int = 25" phoneshell/bench/schema.py
grep -n "max-turns\|error_max_turns\|timed_out = True" phoneshell/bench/runner.py -A2 -B2
# confirm there is no rollback, checkpoint or restore concept anywhere in the codebase
grep -rniE "rollback|checkpoint|restore_state" phoneshell/ || echo "no matches"
# RIR's own trigger rule and reported numbers
curl -s https://arxiv.org/abs/2609.18304 | grep -i "rollback\|percentage point"
# full method and Table 1: https://arxiv.org/html/2609.18304v3What would prove this wrong
We claim phoneshell has no mechanism that preserves any state from a failed attempt or resumes execution from one. That is wrong if a future version of bench/runner.py or actions.py retains a Snapshot across actions, or if any task file gains a field for a partial-credit or retry state. We also claim RIR's reported gains, while real on ALFWorld, ScienceWorld and GAIA, have not been shown to transfer to GUI or mobile agents. That claim is falsified the moment RIR, or a method built on its rollback-boundary framing, is evaluated on a mobile GUI benchmark (iOSWorld, AndroidWorld, DAC-Bench or phoneshell's own 80 tasks) and the result is published. We will treat either event, checked by 31 March 2027, as grounds to revisit this comparison.
Sources
- Yu, Yao, Li, Wang, Wu, Rollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agents, arXiv:2609.18304, submitted 16 September 2026, revised 22 September 2026 (v3), read 15 October 2026.
- Full text and Table 1: arxiv.org/html/2609.18304v3.
- Blolabel, phoneshell:
phoneshell/config.py,phoneshell/bench/schema.py,phoneshell/bench/runner.py, read 15 October 2026.