Diverse experience and the right recipe enable generalist improvement.

MolmoSpaces robot manipulation scene: Training: box
Training: box
MolmoSpaces robot manipulation scene: New scene: box
New scene: box
MolmoSpaces robot manipulation scene: New object: remote control
New object: remote control

Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

Michal Korniak1,* Marco Bagatella1,2,* Ralf Römer3,* Michal Nauman4,5 Yarden As1 Pranav Vignesh6 Manan Agarwal6 Leander Diaz-Bone1,2 Piotr Zalewski5 Mikołaj Czarnecki4 Lenart Treven1 Shashwat Saxena6 Sreyas Venkataraman6 Sarvesh Patil6 Zhiyuan Zhou7 Daniel Marta1 Oier Mees1,8,† Nur Muhammad Mahi Shafiullah7,† Carmelo Sferrazza7,† Bhavya Sukhija1,† Angela P. Schoellig3,† Andreas Krause1,† Max Simchowitz6,9,‡ Andrew Wagenmaker7,10,‡

*Co-first authors, order by coin flip†Co-principal investigators‡Lead principal investigators

Author affiliations
  1. 1 ETH Zurich
  2. 2 Max Planck Institute for Intelligent Systems, Tübingen
  3. 3 Technical University of Munich
  4. 4 University of Warsaw
  5. 5 Nomagic
  6. 6 Carnegie Mellon University
  7. 7 University of California, Berkeley
  8. 8 Microsoft
  9. 9 Amazon FAR
  10. 10 Physical Intelligence
Paper Code Results BibTeX

TL;DR

Post-training of generalist policies often specializes them to a narrow set of tasks and reduces performance elsewhere. We study whether learning from experience across diverse tasks can improve a single policy on both its training tasks and related tasks it has not seen during post-training.

We show that with a sufficiently broad task distribution, critic-guided reinforcement learning (BON, OGPO) improves performance on unseen scenes and objects in MolmoSpaces. Filtered supervised fine-tuning (FSFT) also improves training-task performance, but generalizes less reliably. Our results identify task diversity and value learning as key ingredients for generalist policy improvement.

Success on unseen tasks rises with task diversity from 1 to 144 training tasks, with OGPO and BoN improving far more than FSFT and surpassing the base policy.
From forgetting to generalization. Success on unseen tasks in MolmoSpaces.

Setup

Overview of the protocol: (1) a generalist policy collects rollouts across training tasks; (2) the experience is pooled in a shared buffer and used for direct policy adaptation (FSFT), critic-guided refinement (BoN), or critic-guided policy extraction (OGPO), repeated for M iterations; (3) the policy is evaluated on training tasks and on held-out tasks in LIBERO-90 and MolmoSpaces.
Generalist improvement and evaluation. (1) A generalist policy collects K rollouts on each training task. (2) The experience is pooled in a shared buffer and used to update the policy directly (FSFT), train a critic that ranks actions (BoN), or train a critic and extract a better policy from it (OGPO). This process is repeated for M iterations. (3) The policy is evaluated on its training tasks and on unseen tasks to test generalization.

Separate task improvement from generalization

MolmoSpaces lets us vary two axes of task diversity: the object to manipulate and the scene it appears in. We scale from 1 to 144 training tasks and evaluate transfer to unseen scenes and unseen objects.

Training tasks

MolmoSpaces robot manipulation scene: Box
Box
MolmoSpaces robot manipulation scene: Box
Box
MolmoSpaces robot manipulation scene: Ladle
Ladle
MolmoSpaces robot manipulation scene: Ladle
Ladle

Unseen scenes

MolmoSpaces robot manipulation scene: Box
Box
MolmoSpaces robot manipulation scene: Ladle
Ladle

Unseen objects

MolmoSpaces robot manipulation scene: Remote control
Remote control
MolmoSpaces robot manipulation scene: Wine bottle
Wine bottle

Results

The first examples of generalization in RL post-training of robot foundation models

Single-task post-training generally leads to forgetting — while we can improve on the task we train on, performance degrades on others. However, as we scale the number of training tasks, critic-based recipes (OGPO and BoN) exhibit a transition from forgetting to generalization. We observe performance gains over the base policy of up to +58 pp for unseen scenes and +39 pp for unseen objects.

MolmoSpaces success rates for OGPO, BoN, and FSFT across 1, 4, 16, 64, and 144 training tasks, evaluated on seen tasks, unseen scenes, and unseen objects.
Value-based improvement generalizes beyond the training tasks. Final success rates on training tasks (left) and held-out tasks (center and right) as the training set grows.

What kind of diversity matters?

Diversifying scenes helps generalize to new scenes and diversifying objects helps generalize to new objects. Only scaling object diversity surpasses the base policy on both axes in this comparison.

To achieve generalization to novel manipulation strategies, it is necessary to train on tasks requiring a diversity of manipulation behaviors, and doing so can transfer to robustness over the scene layout and appearance.

Generalization when scaling scene diversity versus object diversity.
Diversification transfers best along its own axis. OGPO success rate when scaling scenes of a single object, or objects with one scene each, evaluated on eight unseen scenes (left) and eight unseen objects (right).

Transfer to unseen objects

Scaling from 64 to 144 training tasks introduces four additional object categories:

Added training categories

  • Salt shaker: actual MolmoSpaces training object
    Salt shaker
  • Soap dispenser: actual MolmoSpaces training object
    Soap dispenser
  • Mug: actual MolmoSpaces training object
    Mug
  • Fruit & vegetables: actual MolmoSpaces training object
    Fruit & vegetables

Transfer to unseen objects · 64 → 144 tasks

Wine bottle

Similar shapes added at 144 tasks

Salt shaker Soap dispenser

Choose an unseen object

BoN improves most on atomizers and wine bottles from 64 to 144 tasks. Both share upright, bottle-like shapes with the newly added salt shakers and soap dispensers. The other evaluated objects have no close shape match among these additions and show little additional improvement. This pattern suggests that BoN transfers across similar object shapes.

For OGPO, the largest gain over the same range is on atomizers (about +16 percentage points). Wine-bottle success remains low, increasing from about 9% to 12%, while bowls, forks, and remote controls reach high success rates. The selected videos illustrate individual outcomes; the accompanying bars and plot show aggregate performance.

Per-object success rates for OGPO, BoN, and FSFT on six unseen object categories. From 64 to 144 tasks, BoN improves most on atomizers and wine bottles; OGPO improves most on atomizers, while its wine-bottle success remains low.
Generalization varies by object and method. Final success rate on each of the 6 unseen objects, none of which appears in any training split, as a function of the number of training tasks.

BibTeX

@misc{korniak2026reinforcementlearningenablesscaling,
  title = {Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement},
  author = {Michal Korniak and
            Marco Bagatella and
            Ralf Römer and
            Michal Nauman and
            Yarden As and
            Pranav Vignesh and
            Manan Agarwal and
            Leander Diaz-Bone and
            Piotr Zalewski and
            Mikołaj Czarnecki and
            Lenart Treven and
            Shashwat Saxena and
            Sreyas Venkataraman and
            Sarvesh Patil and
            Zhiyuan Zhou and
            Daniel Marta and
            Oier Mees and
            Nur Muhammad Mahi Shafiullah and
            Carmelo Sferrazza and
            Bhavya Sukhija and
            Angela P. Schoellig and
            Andreas Krause and
            Max Simchowitz and
            Andrew Wagenmaker},
  year = {2026},
  eprint = {2611.06905},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2611.06905}
}

Acknowledgements

This work was supported under project IDs 143, 037 and 0220 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure.

↑

Figure

Scroll to explore the figure. Press Escape or Close to return.