Beyond Monotonic Progress:
Retry-Supervised Value Learning for Robot Imitation

Xinyao Qin*†1, Junjie Lu*†2, Kaixin Wang3, Chuheng Zhang3, Sinjae Kang4, Kimin Lee4,5, Min Xu2, Bin Liang1, Jun Yang1, Li Zhao3
1Tsinghua University    2University of Technology Sydney    3Microsoft Research    4KAIST    5Config
* Equal contribution.   † Work done at Microsoft Research.
10th Conference on Robot Learning (CoRL 2026)
Paper Video
ReTVL overview showing an example retry trajectory, value curves, and retry-centered preference supervision.

ReTVL turns sparse retry keypoints into local value supervision, learning when execution degrades and recovers so that imperfect demonstrations can be used more selectively for policy learning.

Video

If the embedded player does not load in your browser, you can also watch the video directly on YouTube.

Abstract

Human demonstrations for robot imitation learning often contain mistakes and corrective behaviors, such as imprecise grasps, object misalignment, unstable contact, and repeated attempts. While these segments are commonly treated as noisy or suboptimal data, they provide valuable evidence about when execution deviates from a desirable path and how task feasibility can be restored.

However, existing reward and value models often rely on monotonic progress assumptions, which capture coarse task advancement but may overlook local execution errors and corrective behaviors in imperfect demonstrations. We propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations by leveraging retry events as sparse supervision.

ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning induced by sparsely annotated retry keypoints. The learned value model is then used to reweight demonstration chunks for downstream behavior cloning, reducing the influence of harmful execution errors while preserving useful corrective behaviors. Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines and improves imitation learning from imperfect demonstrations.

Method

ReTVL model architecture and training objectives.
Global progress calibration

An absolute progress objective anchors the global value scale on successful and failed trajectories.

Retry-induced preferences

Sparse retry keypoints induce local pairwise orderings that capture mistake-related value drops and recovery.

Value-guided policy learning

The learned value signal reweights action chunks, suppressing harmful segments while retaining useful corrections.

Results

Value Estimation

Value estimation comparisons across Pick up Spoon, Fold Towel, and Open Drawer.

ReTVL captures localized value drops around retry events and rebounds after correction more clearly than progress-based baselines.

Representative held-out trajectories showing the evolution of predicted values around retry events.

Downstream Policy Learning

Standard BC 41%
→
RECAP-BC 63%
→
ReTVL-BC 80%
+38.75 pp vs. Standard BC

Average success rate across four real-robot tasks, with 20 evaluation trials per task.

Why does ReTVL help?

Better filtering of harmful actions from mixed-quality demonstrations

We use the trained value model to score action chunks in the mixed-quality policy-training dataset, and convert predicted value improvements into behavior-cloning weights.

Compared with progress-based weighting, ReTVL is more sensitive to local execution mistakes and assigns much lower weights to harmful pre-retry actions, while preserving useful corrective behavior.

0.07–0.18 average pre-retry weights with ReTVL across the four tasks, versus 0.45–0.66 with RECAP-BC.
Pre-retry action weights for Standard BC, RECAP-BC, and ReTVL-BC across four tasks.

Qualitative Policy Comparisons

Standard BC, RECAP-BC, and ReTVL-BC are shown from left to right.

Citation

@article{qin2026beyond,
  title={Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation},
  author={Qin, Xinyao and Lu, Junjie and Wang, Kaixin and Zhang, Chuheng and Kang, Sinjae and Lee, Kimin and Xu, Min and Liang, Bin and Yang, Jun and Zhao, Li},
  journal={arXiv preprint arXiv:2606.24633},
  year={2026}
}