Vector Indexing on Object Storage
The practice of building and querying vector indexes over embeddings derived from data stored in S3.
Summary
The practice of building and querying vector indexes over embeddings derived from data stored in S3.
This topic connects the LLM side of the index to the storage side. Embeddings are generated from S3-stored content, indexed for similarity search, and the results point back to S3 objects.
- Vector indexes are not a replacement for structured queries. They answer "what's semantically similar?" not "what matches this predicate?"
- Storing vector indexes on S3 (e.g., LanceDB) is viable but query latency is higher than dedicated vector databases with in-memory indexes.
scoped_toObject Storage, S3 — vectors are derived from and point to S3 data- LanceDB
scoped_toVector Indexing on Object Storage — S3-native vector database - Embedding Model
scoped_toVector Indexing on Object Storage — produces the vectors - Hybrid S3 + Vector Index
scoped_toVector Indexing on Object Storage — the architectural pattern - Embedding Generation
scoped_toVector Indexing on Object Storage — the capability that feeds vectors
Definition
The practice of building and querying vector indexes over embeddings that are derived from data stored in S3.
Semantic retrieval (finding content by meaning) requires vector representations. When the source data lives in S3, the vector index must bridge the gap between unstructured storage and structured similarity search.
Recent developments
- Amazon S3 Vectors GA — 2B vectors per index, 40× scale of preview. AWS announced GA in December 2025 with a "storage-first" architecture that decouples compute from storage; up to 2 billion vectors per index and elastically scaling to 10,000 vector indexes per vector bucket. Reduces TCO by up to 90% for large-scale RAG. Per AWS What's New — S3 Vectors GA.
- "Storage-first" RAG architecture established as a 2026 category. InfoQ's January 2026 coverage names "storage-first" as the new architectural pattern for RAG — vectors live in object storage, compute pools elastically attach to query them. Per InfoQ — S3 Vectors GA storage-first.
- Multiple competing implementations: LanceDB, Turbopuffer, Zilliz Cloud all use the same model. The pattern isn't AWS-only — LanceDB stores vector indices as objects, Turbopuffer runs a vector DB on top of S3, Zilliz Cloud adopts elastic multi-tiered designs (memory + SSD + object/S3) under one architecture. Per arXiv 2601.01937 — Vector Search for the Future.
- Trillion-scale shift drives the architecture. As vector data approach trillion scale, the static memory–SSD architecture faces cost–performance trade-offs that motivate the multi-tier object-storage-backed design. Per arXiv 2601.01937.
- AWS data-economics recalibration framing. ComputerWeekly's analysis links S3 Vectors, S3 Batch, and Intelligent-Tiering together as a coordinated AWS economic move recalibrating the per-GB cost frontier for AI-shaped data. Per ComputerWeekly — AWS recalibrates data economics.
Connections 30
Outbound 2
scoped_to2Inbound 28
scoped_to26used_by1Resources 3
AWS Architecture Blog describes a production-grade 1B+ vector search solution built on LanceDB with S3 as the storage layer.
LanceDB documentation for this serverless vector database built on the Lance columnar format, designed for S3-native storage.
Milvus is the leading open-source vector database with S3-backed storage support; its architecture docs explain how vector indexes are persisted to object storage.