Rollout-Level Replay Buffers
A distributed storage pattern for RL post-training of LLMs that persists individual rollouts and reasoning trajectories to secondary storage — prioritized by advantage, bounded by staleness — so pipelines recycle high-value experiences instead of regenerating them.
Summary
A distributed storage pattern for RL post-training of LLMs that persists individual rollouts and reasoning trajectories to secondary storage — prioritized by advantage, bounded by staleness — so pipelines recycle high-value experiences instead of regenerating them.
The training-side counterpart to inference memory. Because retaining trajectories in VRAM causes OOM, the pattern streams buffers to NVMe and object storage, making S3 the persistent backend of the RL loop.
- This is not classical experience replay — LLM policies drift fast, so it stores/samples per-rollout and strictly bounds staleness, or training destabilizes.
- The compute saving (regeneration can eat 80%+ of the GPU budget) is the whole motivation.
- Rollout-Level Replay Buffers
depends_onObject Storage — durable rollout persistence - verl Hybrid Replay Buffer
extendsit (production implementation) - Rollout Routing Replay (R3)
extendsit for MoE routing stability
Definition
A distributed storage pattern used in reinforcement-learning post-training of LLMs that persists individual model rollouts and reasoning trajectories to secondary storage, letting training pipelines recycle high-value experiences and cut the cost of continuous on-policy data generation.
Dominant RL post-training methods (e.g. GRPO) are sample-inefficient — generating fresh trajectories can consume 80%+ of the post-training GPU budget. Classical experience replay struggles because policies drift and stored rollouts go stale. This pattern stores and samples per-rollout (not per-group), prioritizes by advantage magnitude, and bounds staleness via age-eviction — streaming buffers to NVMe/object storage instead of scarce VRAM.
Persisting and resampling RL rollouts during LLM post-training, trading minor staleness for large compute savings, surviving training restarts.
Recent developments
- Advantage-prioritized, staleness-bounded replay. Stores individual rollouts, prioritizes replay by per-rollout advantage, and caps staleness via age-eviction. Per arXiv:2606.04560 — Rollout-Level Advantage-Prioritized Experience Replay for GRPO.
- Hybrid VRAM→NVMe→object-storage placement. Because retaining trajectories in VRAM causes OOM, buffers stream to local NVMe and distributed object storage (S3/HDFS). Per arXiv:2604.08706 — Efficient RL Training for LLMs with Experience Replay.
Connections 6
Outbound 4
scoped_to1depends_on1related_to1Inbound 2
Resources 2
The advantage-prioritized, staleness-bounded experience-replay design for GRPO — the core mechanism and why per-rollout replay beats group replay for drifting LLM policies.
Efficient RL training with experience replay — the VRAM→NVMe→object-storage placement argument and the staleness/compute trade-off.