Standard

RDMA (RoCE v2 / InfiniBand)

A network transport protocol for direct memory-to-memory data transfer between machines, bypassing the operating system kernel and CPU for minimal latency and maximum throughput. In early 2026, NVIDIA shipped RDMA client and server libraries for S3-compatible storage as part of the CUDA Toolkit, marking the transition from niche technical preview to standard "AI Factory" infrastructure.

9 connections2 resources2 posts

Summary

What it is

Where it fits

RDMA is the high-performance network fabric used by storage systems that need microsecond-level access. Object storage systems serving AI/ML workloads use RDMA to achieve storage access times that approach local NVMe, enabling GPU-direct data paths. The NVIDIA CUDA Toolkit integration means GPU clusters can now access S3-compatible storage over RDMA without custom driver work.

Misconceptions / Traps

RDMA requires specialized network infrastructure. RoCE v2 works on lossless Ethernet (requires PFC/ECN configuration); InfiniBand requires dedicated switches and HCAs.
RDMA performance is highly sensitive to network configuration. Incorrect QoS, PFC, or ECN settings cause performance worse than standard TCP.
The NVIDIA CUDA Toolkit RDMA libraries target S3-compatible storage specifically; not all object stores support the required RDMA transport yet.

Key Connections

enables RDMA-Accelerated Object Access — the transport protocol for microsecond object access
enables GPU-Direct Storage Pipeline — direct storage-to-GPU data path
scoped_to Object Storage — underlying transport for high-performance storage

Definition

What it is

Network transport protocols enabling Remote Direct Memory Access — transferring data directly between application memory on different servers without involving the CPU or OS kernel, achieving microsecond-level latency. In early 2026, NVIDIA released RDMA client and server libraries for S3-compatible storage as part of the CUDA Toolkit, transitioning RDMA from niche technical preview to a standard component of "AI Factory" infrastructure.

Why it exists

HTTP/TCP-based S3 access introduces millisecond-scale latency. For internal object storage data paths (inter-node replication, erasure coding reconstruction), RDMA eliminates protocol overhead, enabling storage fabric performance closer to local memory access.

Primary use cases

High-performance inter-node replication, erasure-coding reconstruction acceleration, AI/ML storage fabric, low-latency data movement within storage clusters, NVIDIA CUDA Toolkit GPU-direct S3 access.

Recent developments

Latest signals

Ethernet (RoCE v2) winning the AI fabric war — ~70% of new deployments. Broadcom's March 2026 earnings confirmed ~70% of new AI infrastructure deployments are choosing Ethernet-based fabrics over InfiniBand. Meta, Microsoft, AWS converging on RoCE v2 for operational reasons (existing Ethernet skills + open-vendor sourcing). Per Rack2Cloud — InfiniBand vs RoCEv2: Why Ethernet Is Winning and NetPilot — RoCEv2 vs InfiniBand: AI Cluster Networking Compared 2026.
Performance gap narrowed: InfiniBand 1–2µs vs RoCE v2 2–5µs. RoCE v2 hits 85–95% of InfiniBand's training throughput for tier-2/3 deployments (256–1,024 GPUs). InfiniBand still wins at frontier scale (>10K GPUs) where the latency delta compounds. Per FirstPassLab — RoCE vs InfiniBand for AI Data Center Networking 2026.
Ultra Ethernet Consortium (UEC) emerging as the third option. UEC builds next-gen Ethernet specifically for AI workloads with built-in reliability that eliminates PFC entirely + adaptive AI-tuned congestion control + native RDMA. Production deployments expected H2 2026 onward. Per Stordis — Ultra Ethernet vs InfiniBand, RoCE and TCP - AI and Medium — From InfiniBand to Ultra Ethernet: Why AI Networks Rethought RDMA.
Lossless Ethernet (PFC + ECN) is the table-stakes RoCE v2 deployment pattern. Production RoCE v2 requires Priority Flow Control + Explicit Congestion Notification across the fabric — getting this configuration right is the load-bearing operational task that separates working RoCE v2 deployments from broken ones. Per Intelligent Visibility — Lossless Ethernet Design Guide for AI Fabrics 2026.
iWARP is dead; RoCE v2 + InfiniBand + UEC are the three options. Intelligent Visibility's RDMA-for-Storage guide explicitly retires iWARP from the production option set — the third option that competed with RoCE v2 and InfiniBand a decade ago has effectively zero new deployments. Per Intelligent Visibility — RDMA for Storage Ethernet: RoCE vs iWARP.
RDMA-for-S3 is the convergence point — cuObject, MinIO AIStor, Cloudian, VAST all wire it via RoCE v2 or InfiniBand. Cross-vendor signal: every major S3-RDMA implementation in 2026 sits on top of RoCE v2 or InfiniBand. The protocol layer is settled; the storage-vendor implementations are where the differentiation lives. Per Distributed AI Fabrics — InfiniBand, RDMA, Lossless Ethernet Strategy Guide.
Huawei's OpenURMA claims 4.37x lower latency and 2.8x higher throughput than RoCEv2 — a fourth fabric option beyond RoCEv2/InfiniBand/UEC. OpenURMA is a clean-room open implementation of Huawei's Unified Bus (UB) protocol, benchmarked against a RoCEv2 reliable-connection baseline. Primary source (arXiv paper), so the numbers are Huawei's own benchmark methodology, not independently reproduced — but it's a credible new entrant to the fabric conversation the current index frames as a three-way RoCEv2/InfiniBand/UEC race. Per OpenURMA: A Clean-Room Open Implementation of the Unified Bus Protocol (arXiv, 2026-05-27).
IBTA Specification Release 2.0 (published July 2025) formalizes XDR speeds and large-radix switching. 200 Gb/s per lane, enabling 800 Gb/s (QSFP) and 1.6 Tb/s (QSFP-DD/OSFP) total link bandwidths, plus support for switches up to 64K ports. This is the spec baseline underneath the XDR InfiniBand and 800G RoCE products now shipping. Per IBTA — What's New in Specification Release 2.0.
RDMA's transport layer has no TLS-equivalent — a security gap the index doesn't currently cover. Academic work (ETH Zurich's ReDMArk, USENIX Security 2021) demonstrated packet injection and unauthorized memory access against Mellanox/Broadcom InfiniBand and RoCE implementations, citing low-entropy rkey generation and sequential QPN enumeration as root causes; a 2026 NDSS paper (JANUS) proposes expressive ACLs specifically because RDMA lacks them natively. CISA's January 2026 vulnerability bulletin lists five separate Linux kernel RDMA CVEs patched the same month. Worth flagging given the current index's push toward RDMA-for-S3 as a performance default — the security tradeoff isn't mentioned. Per ReDMArk (USENIX Security 2021), JANUS (NDSS 2026), and CISA Bulletin SB26-020.
TOP500 supercomputer share tells a different story than the hyperscaler-adoption numbers already in the index. By port count, InfiniBand holds 47.8% of TOP500 systems vs RoCE's 39%; by total port bandwidth, RoCE leads 48.5% to InfiniBand's 39.2%. This is a narrower, HPC-specific metric — distinct from the "~70% of new AI deployments choose Ethernet" hyperscaler figure already cited. Tertiary source, but the two numbers (deployment count vs bandwidth share) are consistent with each other. Per cloudswit.ch — RoCE vs InfiniBand technical comparison.
DCQCN is the actual congestion-control algorithm behind RoCEv2's "lossless" claim — PFC and ECN alone are the backstop, not the mechanism. Data Center Quantized Congestion Notification (DCQCN) does the real work: NIC-side rate-based congestion reaction driven by ECN marking, with Priority-based Flow Control as the last-resort pause signal when DCQCN can't shed load fast enough. This node's existing "PFC + ECN" framing names the guardrails but skips the algorithm actually doing the throttling. Per Ubuntu — What Is RDMA over Converged Ethernet (RoCE)?.
NVIDIA Spectrum-4 switches hit 2-3µs end-to-end RoCEv2 latency when PFC/ECN/DCQCN are configured correctly — a concrete data point at the low end of the 2-5µs range this node already cites, naming the switch silicon that gets there. "Configured correctly" is the load-bearing condition: misconfigured PFC is the standard failure mode (pause-storm cascades that stall the fabric), consistent with this node's existing operational-risk framing. Per Ubuntu — What Is RDMA over Converged Ethernet (RoCE)?.
UEC 1.0 is no longer "emerging" — the spec shipped June 11, 2025, and UEC-capable silicon is shipping (September 2026 update). The Ultra Ethernet Consortium's 1.0 specification introduces the UET transport with sender-based congestion control and multi-path packet spraying, and Broadcom's Tomahawk 6 (102.4 Tbps) is shipping UEC-capable hardware — the "production deployments expected H2 2026 onward" framing earlier in this node is now the live present, tracked in the dedicated Ultra Ethernet (UEC) node. Per UEC — Specification 1.0 launch.
Dell'Oro forecast: Ethernet surpasses InfiniBand in AI back-end networks by 2027 (independent-methodology market forecast). The market-research trajectory behind the "~70% of new deployments choose Ethernet" operational figure this node already carries — cost efficiency, supply-chain diversity, and vendor-lock-in avoidance are the cited drivers, with UEC positioned as Ethernet's endgame for AI. Per Nokia — the future of AI networking with Ultra Ethernet Consortium (citing Dell'Oro).
Provenance note on the "RoCE v2 hits 85–95% of InfiniBand throughput" figure this node carries. A September 2026 primary-source sweep found no independent measurement attributable for that exact percentage — the figure traces to tertiary comparison content (FirstPassLab), not a published methodology. The architectural premise is validated by vendor behavior (NVIDIA's own Spectrum-X800 Ethernet platform ships RoCEv2 alongside InfiniBand at CoreWeave-class deployments, at up to 1.6 Tb/s backend bandwidth per GPU in multi-rail topologies), but treat the 85–95% number as directional, not measured. Per NVIDIA — networking switch announcements. Sources: NetPilot — RoCEv2 vs InfiniBand AI Cluster Networking 2026 · FirstPassLab — RoCE vs InfiniBand 2026 · Rack2Cloud — Ethernet Winning the AI Fabric War · Stordis — Ultra Ethernet vs InfiniBand, RoCE, TCP · Intelligent Visibility — Lossless Ethernet Design Guide 2026