NVMe-oF / NVMe over TCP

Summary

What it is

A protocol family for accessing NVMe storage devices over network fabrics (RDMA, TCP, Fibre Channel), enabling disaggregated flash storage with near-local access latency.

Where it fits

NVMe-oF enables the high-performance storage tier beneath S3. Object storage systems like VAST Data and MinIO can use NVMe-oF to access disaggregated flash arrays with microsecond-level latency — eliminating the local-disk constraint for high-performance object storage.

Misconceptions / Traps

NVMe over TCP is not the same as NVMe over RDMA. TCP has higher latency (tens of microseconds vs single-digit microseconds) but works on standard Ethernet without special NICs.
NVMe-oF disaggregates storage from compute but introduces network reliability as a storage dependency. Network partitions become storage failures.

Key Connections

enables NVMe-backed Object Tier — the protocol for accessing disaggregated flash
scoped_to Object Storage — underlying transport for high-performance object stores

Definition

What it is

A transport protocol specification for accessing NVMe storage devices over network fabrics (TCP, RDMA, Fibre Channel), enabling disaggregated all-flash storage with near-local-disk latency.

Why it exists

Object storage backends ultimately write to physical storage. NVMe-oF allows the flash tier to be disaggregated from compute while maintaining the low-latency characteristics of local NVMe, enabling flexible all-flash object storage architectures.

Primary use cases

Disaggregated flash storage backends for object stores, high-performance storage networking, remote NVMe access for AI/ML workloads.

Recent developments

Latest signals

Two remote-kernel-panic CVEs land in Linux NVMe/TCP (CVE-2026-22998, CVE-2026-23112). The CVE-2026-22998 flaw sits in the kernel's nvmet_tcp_build_pdu_iovec(): an earlier bounds-checking patch never validated that command data structures were initialized, so an unauthenticated attacker who can reach the target can complete the ICREQ/ICRESP handshake and send a malformed H2C_DATA PDU before any CONNECT command — the kernel dereferences uninitialized pointers and panics. Because NVMe/TCP runs on ordinary Ethernet with none of Fibre Channel's physical segregation, the practical guidance shifted: patch immediately, keep NVMe/TCP targets hermetically segregated from application traffic, and weigh RoCE/RDMA transports with hardware-level isolation for AI storage fabrics — the "TCP is the easy transport" default just acquired a hardening tax. Per NVD — CVE-2026-22998, Red Hat advisory, and GHSA-xjx5-8v38-q6gh.
Windows Server vNext (Build 29558) adds native NVMe-oF support — both NVMe/TCP and RDMA. Microsoft joining the initiator side broadens NVMe-oF beyond the Linux-only deployment story and signals the transport is now assumed infrastructure for AI storage fabrics.
State of NVMe-oF in 2026 — wholesale migration didn't happen, but TCP transport quietly won. The WWT industry survey "The State of NVMe-oF in 2026" reports that the predicted across-the-board NVMe-FC → NVMe-oF refresh did not occur — most enterprises are running mixed estates. What did happen is a clear NVMe over TCP win on greenfield: the spec's 2021 ratification + 2022 array-vendor GA cycle landed and TCP-based deployments are now the default new-build choice. Customer attitudes split into "wait and see" (FCoE-burned), "all-in on TCP" (greenfield AI infra), and "FC for legacy, TCP for new" (the pragmatic majority).
NVMe-oF for AI infrastructure — disaggregation of GPU compute from flash is real. Industry analysis "The Promise, the Pain, and What Actually Works" confirms the architectural shift: GPU pools and flash JBOFs scale independently across NVMe-oF (TCP or RDMA) fabrics, with any GPU node addressing any flash shelf. The honest pain points: TCP adds latency nobody budgets for, RDMA requires network-engineering depth, and half the industry deploys NVMe-oF without understanding what they're buying. For AI training infrastructure specifically, the disaggregation pattern is now the default new-build architecture; the question is RDMA vs TCP at the transport layer, not whether to use NVMe-oF at all.
Lightbits Labs records 3× YoY growth on NVMe/TCP standardization (January 2026). Lightbits — inventor of the NVMe over TCP storage protocol — closed 2025 with a 3× year-over-year increase in software purchases, 2× growth in average deal size, and first-time deployments of 4× greater capacity than the prior year. Customer concentration is in financial services, e-commerce, neo-clouds, and CSPs — the workloads where AI training and real-time analytics are pushing legacy SAN/HCI past their predictable-performance limits. Quoted differentiator: 5× greater hardware efficiency vs Ceph Storage under equivalent workloads. Treat the numbers as vendor-attested, but the pattern (legacy SAN displacement by SDS over NVMe/TCP) is corroborated across analyst surveys.
NVMe over TCP Transport Specification reaches Revision 1.2 (ratified August 2025). The revision formalizes optional inline data integrity (DIGEST) and online TLS negotiation for encrypted transport sessions, on top of the existing queue/capsule/data-delivery mapping. Per NVM Express — NVMe over TCP Transport Specification.
Standalone NVMe-oF specification retired — content merged into the NVMe Base spec plus separate per-transport specs. NVM Express confirms no further development is planned on the standalone NVMe-oF spec; PCIe, TCP, and RDMA transport bindings now each live in their own dedicated specification documents. Per NVM Express — NVMe over Fabrics (oF) Specification, historical reference.
Three new NVMe/TCP hardening TPs landed with the August 2025 Base Spec Rev 2.3 / TCP Transport Rev 1.2 cycle: TP8018 (TLS updates), TP8025 (NVMe-oF security configurations), TP8028 (rapid path failure recovery). Per NVM Express — Changes in NVM Express Specifications.
NVIDIA DOCA documents zero-CPU NVMe-oF target offload on ConnectX-5 and above. The HCA processes IO requests and routes them directly to the NVMe PCIe device via peer-to-peer PCI transfers, with no host CPU utilization outside connection setup and error handling. Per NVIDIA DOCA — NVMe-oF docs. Sources: The State of NVMe-oF in 2026 (wwt.com) · NVMe-oF: The Promise, the Pain, and What Actually Works (muratkarslioglu.com) · Lightbits Labs Delivers Record Growth as Organizations Standardize on NVMe over TCP (Business Wire via Yahoo Finance)