NVMe-oF / NVMe over TCP

Summary

What it is

A protocol family for accessing NVMe storage devices over network fabrics (RDMA, TCP, Fibre Channel), enabling disaggregated flash storage with near-local access latency.

Where it fits

NVMe-oF enables the high-performance storage tier beneath S3. Object storage systems like VAST Data and MinIO can use NVMe-oF to access disaggregated flash arrays with microsecond-level latency — eliminating the local-disk constraint for high-performance object storage.

Misconceptions / Traps

NVMe over TCP is not the same as NVMe over RDMA. TCP has higher latency (tens of microseconds vs single-digit microseconds) but works on standard Ethernet without special NICs.
NVMe-oF disaggregates storage from compute but introduces network reliability as a storage dependency. Network partitions become storage failures.

Key Connections

enables NVMe-backed Object Tier — the protocol for accessing disaggregated flash
scoped_to Object Storage — underlying transport for high-performance object stores

Definition

What it is

A transport protocol specification for accessing NVMe storage devices over network fabrics (TCP, RDMA, Fibre Channel), enabling disaggregated all-flash storage with near-local-disk latency.

Why it exists

Object storage backends ultimately write to physical storage. NVMe-oF allows the flash tier to be disaggregated from compute while maintaining the low-latency characteristics of local NVMe, enabling flexible all-flash object storage architectures.

Primary use cases

Disaggregated flash storage backends for object stores, high-performance storage networking, remote NVMe access for AI/ML workloads.

Recent developments

Latest signals

State of NVMe-oF in 2026 — wholesale migration didn't happen, but TCP transport quietly won. The WWT industry survey "The State of NVMe-oF in 2026" reports that the predicted across-the-board NVMe-FC → NVMe-oF refresh did not occur — most enterprises are running mixed estates. What did happen is a clear NVMe over TCP win on greenfield: the spec's 2021 ratification + 2022 array-vendor GA cycle landed and TCP-based deployments are now the default new-build choice. Customer attitudes split into "wait and see" (FCoE-burned), "all-in on TCP" (greenfield AI infra), and "FC for legacy, TCP for new" (the pragmatic majority).
NVMe-oF for AI infrastructure — disaggregation of GPU compute from flash is real. Industry analysis "The Promise, the Pain, and What Actually Works" confirms the architectural shift: GPU pools and flash JBOFs scale independently across NVMe-oF (TCP or RDMA) fabrics, with any GPU node addressing any flash shelf. The honest pain points: TCP adds latency nobody budgets for, RDMA requires network-engineering depth, and half the industry deploys NVMe-oF without understanding what they're buying. For AI training infrastructure specifically, the disaggregation pattern is now the default new-build architecture; the question is RDMA vs TCP at the transport layer, not whether to use NVMe-oF at all.
Lightbits Labs records 3× YoY growth on NVMe/TCP standardization (January 2026). Lightbits — inventor of the NVMe over TCP storage protocol — closed 2025 with a 3× year-over-year increase in software purchases, 2× growth in average deal size, and first-time deployments of 4× greater capacity than the prior year. Customer concentration is in financial services, e-commerce, neo-clouds, and CSPs — the workloads where AI training and real-time analytics are pushing legacy SAN/HCI past their predictable-performance limits. Quoted differentiator: 5× greater hardware efficiency vs Ceph Storage under equivalent workloads. Treat the numbers as vendor-attested, but the pattern (legacy SAN displacement by SDS over NVMe/TCP) is corroborated across analyst surveys.