Lance Format

A modern columnar data format optimized for random access, vector search, and high-throughput reads from object storage. Designed as an alternative to Parquet for AI/ML workloads, providing up to 60× faster random access on NVMe vs default Parquet configurations while maintaining strictly comparable sequential scan speeds. Arrow-native, zero-copy, written in Rust. Optimized for the access patterns that AI dataloaders, vector retrieval engines, and embedded analytical queries actually use — not the bulk-scan pattern that Parquet was designed for.

Apache Parquet, designed over a decade ago for distributed analytical processing, fails AI workloads on three structural axes. (1) Parquet encodings are not sliceable — retrieving a single image tensor or contextual chunk forces loading, decompressing, and decoding an entire page or row group. (2) Wide columns with thousands of fields make optimal row-group sizing mathematically impossible, leading to memory bloat and unpredictable read latency. (3) Heavy chatty network behavior — excessive HTTP requests when interacting with remote object stores. Lance addresses each: adaptive structural encodings for random access, sophisticated metadata management for wide-schema feature stores, multi-level shredding for nested validity, and dedicated Blob semantics for large binary payloads.

Vector storage and similarity search on S3, AI/ML retrieval workloads requiring random access, embedding store format for LanceDB, multimodal feature stores combining structured + dense-vector + binary-blob columns, autonomous-vehicle / robotics ML training where sensor data and embeddings co-reside, persistent agentic memory layers.