Standard

WebDataset

The tar-shard convention (same-basename files = one sample, numbered shards, brace notation) plus a BSD-3 Python library that streams those shards into PyTorch DataLoaders from local disk or any S3-compatible endpoint.

6 connections 4 resources 1 post

Summary

What it is

Where it fits

It is the storage-format answer to the data-loading bottleneck: sequential tar reads keep GPUs fed from object storage at near-wire speed, where per-file random reads collapse. Choose shard formats (WebDataset tar, or MosaicML MDS) when samples are large binary blobs — images, audio, video, npy tensors — and the job is sustained streaming for training. Parquet/JSONL remain the better fit for tabular/text corpora where you also want columnar analytics, schema, and cheap random access; small SFT text sets don't need shards at all.

Misconceptions / Traps

WebDataset shuffling is approximate (shard shuffle + a fixed-size sample buffer), not a global shuffle — and unlike MDS it has no built-in deterministic ordering across varying GPU counts or mid-epoch resumption.
The "format" is just POSIX tar with naming rules; nothing needs converting in, and tar can extract it back out. Lock-in lives in the pipeline code, not the data.
GitHub releases lag PyPI badly (v0.3.2 tag vs 1.0.2 on PyPI) — check PyPI, not the releases tab, for the current version.

Key Connections

implements Training Data Streaming from Object Storage — the canonical open format for it; pipe: URLs make any S3-compatible store a training source.
solves Data Loading Bottleneck — sequential shard reads vs. millions-of-small-files random I/O.
alternative_to Apache Parquet — shards win for large media blobs and pure sequential training throughput; Parquet wins for tabular/text plus analytics.

Definition

What it is

A dataset format convention plus a Python I/O library for streaming training data as sharded POSIX tar archives. Files that share a basename inside a tar (e.g. `0001.jpg` + `0001.json`) form one training sample; shards are numbered `dataset-000000.tar … dataset-012345.tar` and addressed with brace notation. The library implements PyTorch's `IterableDataset`, so a DataLoader can consume shards streamed straight from disk, an HTTP server, or any S3-compatible bucket via `pipe:` URLs (`pipe:curl -s https://…` or an `aws s3 cp - ` pipe) — no data conversion, no volume plugins.

Why it exists

Training on millions of small files murders filesystems and object stores alike — random per-file reads can't feed multi-GPU nodes that need aggregate GB/s. Packing samples into ~100MB–1GB tar shards turns all I/O into large sequential reads: the PyTorch team reports roughly 10x I/O improvement on hard drives over file-based random access, and near-hardware-limit throughput (~150 MB/s per rotational drive) when scaling to hundreds of clients against object storage. For self-hosters this is the practical way to train from a MinIO/S3 bucket without copying the dataset to local NVMe first.

Recent developments

Latest signals

webdataset 1.0.2 published to PyPI (June 19, 2025). The library crossed the 1.0 line; BSD-3-Clause, Python >=3.10, maintained by Thomas Breuel. GitHub tagged releases lag PyPI (last tag v0.3.2, Feb 2025). Per PyPI.
Hugging Face Hub documents WebDataset as a first-class dataset format (current docs). HF recommends it specifically for multimodal (image/audio/video) datasets, describes ~1GB shards as typical, and shows streaming Hub-hosted shards through pipe:curl with auth headers. Per HF Hub docs.
MosaicML Streaming 0.13.0 shipped (July 15, 2025) — the main alternative stays active. Its MDS shard format adds what WebDataset's sequential model lacks: low-latency random access, deterministic shuffling independent of GPU count, and mid-epoch resumption; backends include S3, GCS, Azure, OCI, and S3-compatibles (R2, Backblaze B2). Databricks' own benchmark claims ~19k vs ~16k img/s over WebDataset on ImageNet (vendor benchmark). Per mosaicml/streaming and PyPI.
v0.2.107 release notes (Sep 27, 2024) added bz2/xz shard compression, a DecodingError exception, and WDS_PYTORCH_WEIGHTS_ONLY for safer tensor loading. Per GitHub releases.