DualPipe

Bidirectional pipeline-parallelism algorithm released as part of DeepSeek's Open Source Week Day 4 (February 2025), explicitly designed for the V3/R1 training stack. Overlaps forward and backward computation-communication phases by orchestrating them in bidirectional streams — while one set of micro-batches is doing forward processing, another set is simultaneously running backward. Each chunk is divided into **four components: attention, all-to-all dispatch, MLP, all-to-all combine**, allowing fine-grained overlap with NVLink communication. PyTorch 2.0+ compatible; integrates into existing training pipelines.

Cross-node expert-parallel MoE training has heavy all-to-all communication overhead — dispatching tokens to experts on remote GPUs, then combining the results, dominates training-step wall-time. Traditional pipeline-parallelism leaves big "pipeline bubbles" while waiting on these all-to-all calls. DualPipe's bet: structurally overlap the communication phases with computation in the *other direction* of the pipeline, so neither GPUs nor NICs sit idle. The result was a key efficiency lever in DeepSeek V3's claimed <$6M training-cost number.

Frontier-MoE training where pipeline bubbles dominate wall-time, large-scale cross-node expert parallelism (V3 uses 16-way PP + 64-way EP + ZeRO-1 DP), distributed training where communication-computation overlap is the gating efficiency factor, and as a reference for any pipeline-parallel training framework adopting bidirectional scheduling.