Rollout Routing Replay (R3)
An RL synchronization mechanism that stabilizes Mixture-of-Experts models by recording the exact expert-routing masks chosen during inference rollout and replaying those decisions in the training forward pass.
Summary
An RL synchronization mechanism that stabilizes Mixture-of-Experts models by recording the exact expert-routing masks chosen during inference rollout and replaying those decisions in the training forward pass.
Where the AI memory fabric meets training correctness: it shows that persisting exact inference execution state (routing masks) — increasingly to object storage — is part of the memory substrate, not just data.
- The instability it fixes is MoE-specific: routers can pick different experts for identical inputs between inference and training, exploding KL divergence.
- It complements, not replaces, replay buffers — the masks ride along in the buffer.
- Rollout Routing Replay (R3)
optimizes_forMixture-of-Experts (MoE) — halves KL divergence - Rollout Routing Replay (R3)
extendsRollout-Level Replay Buffers - Related to DeepSeekMoE
Definition
A reinforcement-learning synchronization mechanism for stabilizing Mixture-of-Experts (MoE) models. It records the exact expert-selection routing masks made during the inference rollout and replays those precise routing decisions during the training forward pass, preventing the policy divergence that destabilizes MoE RL.
MoE routers can pick different experts for identical inputs between inference and training, creating large token-probability deviations and KL divergence that can collapse RL training. R3 forces training gating weights to use the masks captured at inference — records that are increasingly persisted in the replay buffer on object storage.
Stabilizing RL post-training of MoE reasoning models, reducing KL divergence between rollout and update phases, persisting exact inference execution state as part of the memory fabric.
Recent developments
- Replaying routing masks halves KL divergence. Computing training gating with inference-time masks cuts KL divergence by ~half, matching dense-model stability. Per Emergent Mind — Rollout Routing Replay (R3).
- Routing records persisted to the replay buffer. R3 saves routing masks into buffers increasingly backed by object storage, tying inference state into the AI memory fabric. Per Emergent Mind — Rollout Routing Replay (R3).
Connections 4
Outbound 4
scoped_to1optimizes_for1extends1related_to1