verl Hybrid Replay Buffer
A scalable persistence architecture for LLM reinforcement-learning workloads in the verl framework — hot data in RAM, cold in local RocksDB, periodic HDFS/S3 checkpointing — so terabytes of rollout data survive failed training runs.
Summary
A scalable persistence architecture for LLM reinforcement-learning workloads in the verl framework — hot data in RAM, cold in local RocksDB, periodic HDFS/S3 checkpointing — so terabytes of rollout data survive failed training runs.
The production-grade implementation of Rollout-Level Replay Buffers, and a concrete case of object storage as the durable backend for the RL post-training loop.
- The S3 tier is for survivability and cross-run reuse, not the hot path — hot rollouts stay in RAM; eviction is write-through to RocksDB first.
- Without this hierarchy, in-memory rollout buffers OOM the GPU and destroy hours of compute.
- verl Hybrid Replay Buffer
extendsRollout-Level Replay Buffers - verl Hybrid Replay Buffer
integrates_withObject Storage — periodic checkpointing for durability - verl Hybrid Replay Buffer
storesCheckpoint/Artifact Lake on Object Storage
Definition
A scalable persistence architecture for LLM reinforcement-learning workloads in the verl framework. It combines an in-memory cache for hot rollouts, local RocksDB instances for cold data, and periodic HDFS/S3 checkpointing for long-term survivability and reuse across training runs.
RL rollouts generate multi-gigabyte states; purely in-memory buffers trigger GPU OOM and destroy hours of compute. verl establishes a hierarchical pipeline — RAM for hot data, write-through eviction to local RocksDB, periodic upload to object storage/HDFS — so rollout data survives failed trials and supports prioritized sampling across terabytes.
Persistent replay buffers for GRPO/RL post-training, prioritized sampling over terabyte-scale rollout data, cross-run reuse of historical rollouts.
Recent developments
- Hierarchical hot/cold buffer with S3 checkpointing. Hot rollouts in RAM, LRU eviction to local RocksDB, periodic upload of historical rollouts to object storage/HDFS. Per verl GitHub — Persistable Replay Buffer (Issue #2584).
- Plugin sampling over persisted data. Custom interfaces (e.g. prioritized-time sampling) operate safely across terabytes backed by S3. Per verl GitHub — RFC persistable replay buffer (Issue #2539).
Connections 4
Outbound 4
scoped_to1extends1integrates_with1Resources 2
The design discussion for verl's persistable replay buffer — the hot-RAM / cold-RocksDB / S3-checkpoint hierarchy and plugin sampling interface for terabyte-scale RL rollout data.
The originating RFC for large-scale rollout-data storage, motivating the move off in-memory buffers to prevent GPU OOM.